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ABSTRACT 

An evaluation of the Research and Development Centers 
and the Regional Educational Laboratories of the Office of 
Educational Research and Improvement (OERI) was conducted by an 
outside analyst brought in in September 1991 by then Assistant 
Secretary Diane Ravitch. The Research and Development Centers have 
been one of the primary sources of federally supported education 
research for nearly 30 years. This analysis of the nature and quality 
of the research they produce indicates that the overall quality of 
the research is mixed. However, excellent educational research has 
been, and can be, produced by these centers. Steps to ensure this are 
summarized. At present, development seems to be emphasized less among 
center activities. A similar look at the types and quality of 
research and development at the regional educational laboratories 
analyzed activities since 1985. This undertaking is more complicated 
than the previous evaluation because the laboratories have not always 
been as explicitly and directly focused on research and development 
as the centers. Over time, their missions and operation have changed 
enough so that OERI would do well to reconsider the role of these 
laboratories and to look into the provision of research-based 
technical assistance that may be available elsewhere. Suggestions for 
improving the laboratory system are given. Appendixes include the 
curriculum vitae of the author, a list of centers and laboratories, 
and a reply to comments on the report by the chair of the 
Organization of Research Centers. Six figures illustrate the 
discussion. (SLD) 
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INTRODUCTION 



Despite the periodic concerns about the educational research 
funded by the federal government , it is almost impossible to find 
any indepth assessments of the quality of that work. In general , 
government employees usually focus on identifying appropriate 
research topics , ensuring a fair and impartial review of the 
proposals , and funding the best proposals • In addition, efforts 
are made to monitor the progress of the grants and contractors — 
which may or may not include inspection of the work underway. The 
procurement process is closely supervised by both program and 
budget specialists within the U.S. Department of Education. But 
much less attention paid to assessing systematically the quality of 
the commissioned work once the research project has been completed. 
Perhaps an implicit assumption often is that if you fund a well- 
designed research project undertaken by a competent scholar, there 
is less need to scrutinize the final product. As a result, there 
have been little, if any, detailed investigations of the quality of 
educational research submitted by federal grantees and contractors. 

The relative lack of systematic assessment of the quality of 
research finally produced is not confined to the U.S. Department of 
Educatior — it tends to reoccur throughout much of the federal 
government. For example, the Office of Adolescent Pregnancy 
Programs in the Department of Health and Human Services often did 
not make much serious use of the final research or evaluation 
reports submitted by grantees or contractors — let alone examine 
carefully and critically the quality of those final products on a 
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regular basis. 1 Given the widespread, though often erroneous, 
perception among many policy makers and the general public that the 
quality of most educational research is particularly weak, it is 

unfortunate that so little has been done to examine and address 

2 

this problem during the past 20-30 years. 

Much of the rather small amount of research done by the U.S. 
Department of Education is funded by the Office of Educational 
Research and Improvement (OERI) — the successor to the National 
Institute of Education (NIE) which was created two decades ago. 
Despite sporadic calls for examining the quality of work funded by 
NIE or OERI, the agency never commissioned a detailed examination 
of the research and development it sponsored. Partly this 
reflected NIE/OERI's lack of concern about the quality of research 
and development and partly it reflected the extremely short tenure 
of most of the politically appointed leaders of that agency. Even 
outside analysts, such as the recent National Academy of Science 

panel to study OERI, have not explored the quality of the work 

3 

funded by that federal agency. 

1 For an analysis of the research at the Office of Adolescent 
Pregnancy Programs (OAPP) in the late-1970s and early 1980s, see 
Maris A. Vinovskis, An»Epidemic" of Adolescen t Pregnancy? Some 
Historical and Policy Perspectives (New York: Oxford University 
Press, 1988). 

2 On the perception of policy makers and the public on the 
quality of educational research, see Carl F. Kaestle, "Everyone^ 
Been to Fourth Grade: An Oral History of Federal R&D in Education." 
Final Report to the Committee on the Federal Role in Education 
Research, National Academy of Sciences (September 1991) . 

3 Richard C. Atkinson and Gregg B. Jackson, eds., Research and 
Education Reform: Roles for the Office of Educational Re search and 
Improvement (Washington, DC: National Academy Press, 1992). 
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Diane Ravitch, an accomplished scholar and an ardent school 
reformer, was appointed the Assistant Secretary for OERI in mid- 
1991. While much of her personal focus in that office was on the 
development of world-class curriculum standards and frameworks, she 
also was deeply concerned about the quality of educational research 
in general — perhaps in part reflecting her recent previous 

experiences in co-directing the National Academy of Education's 

4 

study, Research and the Renewal of Education , 

Diane Ravitch contacted me in September 1991 and inquired 
whether I might be available and willing to come to OERI on a 
temporary excepted service appointment to examine the nature and 
quality of work supported by that agency and to make suggestions 
for how to improve it. She was familiar with some of my scholarly 
work since we are both historians of education and had met at 
several academic conferences; but we had never established a close 
working relationship prior to my employment at OERI. My training 
in the social sciences as well as my quantitative and qualitative 
work in American history provided me with a broad background to 
examine and evaluate a variety of kinds of research. Moreover, she 
was anxious to bring in someone outside the mainstream of the field 
of education who might not hesitate to criticize the commissioned 
research if necessary. What she did not realize at the time was my 
previous experiences in working on the staff of the U.S. House 
Select Committee on Population in 1978 and serving as a frequent, 



National Academy of Education, Research and th e Renewal of 
Education (Stanford: National Academy of Education, 1991) . 
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long-term consultant to the Office of Adolescent Pregnancy Programs 

(OAPP) and to the Office of Family Planning (OFP) in the Department 

of Health and Human Services (DHHS) from 1981-1985. Working for 

the U.S. Congress and DHHS provided me with invaluable experiences 

5 

for understanding the tasks at OERI. 

Originally I was scheduled to join OERI full-time in January 
1992, but delays in completing the paperwork meant that I served as 
a part-time consultant for the first six months. My official title 
was Research Advisor to the Office of the Assistant Secretary (OAS) 
which permitted me to range broadly across programs within OERI and 
pursue different research-related assignments. The primary and 
initial task assigned to me was to analyze the nature and quality 
of research and development funded by OERI and to make specific 
recommendations for improving them. Diane Ravitch did not tell me 
what particular aspects of OERI research and development to study 
or how to proceed. 

Prior to joining OERI, I had only a vague and incomplete 
understanding of the operations of that office or of the system of 
centers and labs that it funded. In order to familiarize myself 
with OERI, I initially reviewed the extensive briefing materials 
assembled for the panel studying OERI for the National Academy of 
Science and began discussions with the staff at OERI. Gradually, 
I developed a preliminary strategy for investigating the research 
and development funded by the office. 

Since a major portion of the research and development funds of 

5 I have included a copy of my resume as appendix A. 
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OERI were expended on the centers and laboratories, this was a 
logical place to begin. Because of my familiarity with the 
academic community, I started with the centers and then turned my 
attention to the laboratories. Afterwards, I tentatively had 
expected to analyze the National Center for Educational Statistics 
(NCES) and then the field- initiated research. Given that the study 
of the centers and laboratories took longer than initially 
anticipated (in part due to other assignments I received) and the 
change in administrations, only the analysis of the quality of 
research and development at the centers and laboratories has been 
completed before returning to the University of Michigan in August 
1993. 6 Therefore, rather than a series of individual studies of 
different sources of research and development at OERI as originally 
envisioned, this report only contains the analysis of the centers 
and laboratories. Hopefully, OERI will continue and complete this 
series of reviews of the quality of research funded by other 
elements in the total OERI program. 

While Diane Ravitch wanted an examination of the nature and 
quality of research and development, she also expected specific 
recommendations on how to improve the overall system as well. As 
the research advisor to OAS, I quickly became involved in working 
with the Office of Research (OR) and the Programs for the 

6 I was appointed for a three-year term at OERI and the change 
in administrations did not affect my employment in the agency. 
However, even prior to the November 1992 presidential election, I 
was called by my colleagues at the University of Michigan to chair 
our Department of History. Unfortunately, I really could not 
refuse that request and therefore decided to return to the 
University of Michigan for the fall term in 1993. 

ER?C g 



Improvement of Practice (PIP) to enhance the quality of research 
and development at the centers and laboratories. Since a 
preliminary draft of the center report was completed in July 1992 
but the one on the laboratories was only finished in January 1993, 
there was more of an opportunity to influence the changes in the 
operation of OR than in PIP. In both cases, however, it was 
expected that I would work closely with the staffs of those 
programs to implement any changes or recommendations that were seen 
as desirable by the leadership of those programs as well as by the 
Assistant Secretary. 

While most of the reports focus on the nature and quality of 
research and development at the centers and laboratories, there are 
also some broader observations and recommendations on the role of 
the centers and laboratories as well. These broader findings flow 
not only from my specific research on the centers and laboratories, 
but from my larger experiences with research-related activities 
within OERI as well as from the writings of other analysts about 
NIE and OERI. Particularly useful were my experiences as the co- 
ordinator of the OERI 3-5 year planning process during 1992 as it 
gave me ample opportunity to work closely with the wide array of 
research and development activities within OERI— not just those 
associated with the centers and laboratories. Thus, while most of 
the analyses and recommendations in the two reports on center and 
laboratories focus on the nature and quality of the research and 
development, there are also some broader suggestions of the role of 
these institutions within the overall OERI context. 
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Finally, I want to acknowledge the extensive assistance and 
encouragement I have received from the staffs at OERI, the centers, 
and the laboratories . While sometimes we have differed over 
interpretations of particular issues or over the recommendations 
made in this report, this has never interfered with the willingness 
of those staffs to provide information and to share their ideas. 
The transition from the Bush to the Clinton Administration did not 
impede the writing of this report at all thanks to the continued 
open and friendly co-operation first from Emerson Elliott, the 
Acting Assistant Secretary, and then from Sharon Robinson, the 
current Assistant Secretary of OERI. While the Assistant 
Secretaries and other staff members at OERI made extensive 
constructive comments and criticisms about the draft reports, none 
of them ever tried to insist upon a particular interpretation or 
attempted to censor any of my findings or recommendations. 



Chapter 1 



ANALYSIS OF THE QUALITY OF WORK PRODUCED BY 
THE OEPI RESEARCH AND DEVELOPMENT CENTERS 



The research and development centers of the Office of 
Educational Research and Development (OERI) have been one of the 
primary sources of federally-supported education research from 
their inception nearly three decades ago. From 1964 through 1991, 
approximately $620 million (in constant 1982-84 dollars) have been 
expended on these centers. 1 Despite the large amount of funds 
that have been spent, almost no effort has been made to analyze the 
research produced — though individuals sometimes have commented on 
its quality and usefulness based upon their general perceptions of 
these institutions • 

This study of the research and development centers of the 
Office of Educational Research and Development (OERI) focuses on 
the nature and quality of the research produced by these 
institutions. As part of that effort, some analysis of their 
budgets and staffing will be made, but little specific attention 
will be paid to an evaluation of their other important activities 
such as dissemination or the impact of their research. 

This analysis begins with an examination of the major trends 
and ciianges in policy toward the research and development centers 



The information on center funding is based upon data from an 
OERI draft document, "Center Funding History, 1964-1992" (October 
1, 1991) . Since that document had some data missing for the early 
years, I have tried to update it with information from other 
sources such as the annual congressional appropriation hearings. 
While the general funding levels and trends are probably correct, 
there may be some minor inaccuracies in the figures for some of the 
R&D centers. The cost of living information is from U.S. 
Department of Commerce index of consumer prices. 



since the early 1960s, Then five current and two former centers 
are examined in considerable detail by analyzing the amount and 
types of research they have conducted, the quality of their 
research products, and the ways in which the staff of OERI 
interacted with these centers. While the necessity of focusing 
attention only on a small number of centers may somewhat limit the 
extent of the generalizations that can be drawn from this study, 
hopefully the issues raised and the recommendations made as the 
result of this investigation will contribute in a modest way toward 
improving the overall quality of the research produced by the 
centers . 

I. Historical Development of Centers 

Although the federal govei unent established the U.S. Office of 

Education (OE) in 1867, it did not become heavily involved in 

research until the 1950s. The Cooperative Research Act of 1954 

(Pub3lc Law 83-531) authorized research and development (R&D) 

activities in universities and state education agencies. Yet the 

total amount of money for federal education research was limited 

and most of the initial grants were quite small. In 1956 

approximately $1 million dollars was appropriated for educational 

research — but two-thirds of it was earmarked for research on the 

2 

education of the mentally retarded. 

2 For a useful summary and discussion of the early federal 

efforts in research, see Richard A. Dershimer, The Federal 

Government and Educational R&D (Lexington, MA: Lexington Books, 



12 

Moreover, there was growing concern that many of the 
individual education research projects were uncoordinated and 
unrelated so that they would not lead to any significant cumulative 
advances in the field. In one of the earliest, comprehensive 
analyses* of the federal educational laboratories and centers, 
Francis Chase characterized these earlier efforts: 

Our older educational institutions were not well 
adapted to provide continuous development based on 
research. Because of this, we have had a history of 
erratic innovation in education. All too often, heralded 
innovations have meant the introduction of partially 
worked out ideas and systems without adequate provisions 
either for continuing refinement or for modification of 
other elements with which the new components must 
interact. The result frequently has been failure to 
achieve the expected benefits and consequent discard of 
theories and technologies before full exploration of 
their usefulness. In other words, we are suffering in 
education not so much from lack of innovation as from 
arrested development. The educational landscape is 
littered with bright ideas which once evoked high hope 
and with technologies and systems imperfectly adapted to 



1976) . 
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educational use. 

The response to this fragmentation of federal education 

research was the creation of the first R&D centers in 1964 under 

the authorization of the Cooperative Research Act and administered 

by the U.S. Office of Education. The passage of Title IV of the 

Elementary and Secondary Education Act (ESEA) of 1965 provided the 

authorization for more R&D centers as well as the creation of new* 

4 

regional educational laboratories. By FY67 a total of 10 R&D 

centers, each affiliated with a major university, had been 

5 

established. 

Each R&D center was expected to have a programmatic focus. 
Ward Mason, Director of the R&D Center Program from 1964 to 1972, 
listed five guidelines that have remained characteristic of the 
program throughout its history: 

Concentration of effort on a significant 
education problem. 

3 Francis S. Chase, "The National Program of Educational 
Laboratories : An Independent Appraisal of Twenty Educational 
Laboratories and Nine University Research and Development Centers 
Conducted Under Contract No. OEC-3-7-001536-1536 w (Washington, DC: 
U.S. Department of Health, Education, and Welfare, 1968), p. 8. 

4 For an analysis of the passage of Elementary and Secondary 
Education Act (ESEA) of 1965, see Stephen K. Bailey and Edith K. 
Mosher, ESEAs The Office of Education Administers a Law (Syracuse, 
NY: Syracuse University Press, 1968) . 

5 U.S. Congress, House, Committee on Education and Labor. 
Study of the United States Office of Education , 90th Congress, 1st 
Session (House Document No. 193: Washington, DC: U.S. Government 
Printing Office, 1967), p. 224. 
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Interrelated projects and activities 
planned to achieve defined objectives 0 

Responsibility for program management 
delegated to center administrators with broad 
responsibility to the Office of Education. 

Interdisciplinary teams recruited from 
throughout the university as required by the 
problems addressed. 

Vertical integration of activities across 
the full range of research , development , and 
dissemination functions . 6 

The early centers were intended to be large , national 

institutions which focused on some education problem area (in the 

initial competitions for R&D Centers , the Office of Education did 

not specify what education problems or issues had to be addressed, 

but left it up to the applicants to decide what might be most 

appropriate) . Among the R&D centers created between 1964 and 1966, 

the average annual funding for the first three years was 

approximately $2.5 million (in constant 1982-84 dollars) — 

considerably larger than the current $780,000 (in constant 1982-84 

dollars) average annual funding for centers created between 1989 
7 

and 1991. Similarly, there has been a substantial reduction in 

6 Ward S. Mason, "Two Decades of Experience with Educational 
R&D Centers" (Washington, DC: National Institute of Education, 
1983) • 

7 

For details of the data on center funding, see footnote one. 
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the average cost per center (in constant 1982-84 dollars) over time 
(see figure 1) . 

Insert figure 1 

If there was general agreement that the early R&D centers 
should be large national institutions organized around some 
education issue or problem, there was less consensus about the 
exact nature of the work that was to be performed, 8 Title IV of 
the Elementary School and Secondary Education Act of 1965 did not 
specify exactly what activities R&D centers were expected to 
undertake. Nor did the influential John Gardner Task Force on 
Education in 1964 provide much concrete guidance for the centers — 
though it did provide much more detail on their expectations for 
the new education laboratories. 9 However, the directives from the 
Office of Education as well as several assessments of the centers 
and laboratories, do provide some clues to the intentions of the 
founders as well as the early office of Education administrators, 
Jerry Walker, who consulted the relevant documents as well as 
interviewed many of the participants, noted that: 

Q 

For an analysis of the activities of education research 
centers in the 1960s, see Sam D. Sieber, Reforming the University: 
The Role of the Social Re search Center (New York: Praeger, 1972) . 
9 

John Gardner, "Report of the President's Task Force on 
Education" (November 14, 1964). Available at the LBJ Presidential 
Library, Austin, Texas. For a very useful analysis of the 
activities of that task force, see Charles Philip Kearney, "The 
1964 Presidential Task Force on Education and the Elementary and 
Secondary Education Act of 1965." Unpub. Ph.D. diss., University 
of Chicago, 1967. 
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FIGURE 1 
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From a review of documents which discuss the origins 
of labs and centers , and a review of subsequent documents 
produced by, or about , labs and centers , it is evident 
that while the initial expectations for their roles and 
functions were obscure and shifting, they did 
differentiate labs from centers in terms of fundamental 
roles and functions.,.. The initial expectations 
appeared, amid the flux and flax from which they 
emanated, to differentiate labs from centers primarily in 
terms of the extension beyond research expected of labs; 
the problem foci expected of both were to be attacked 
more by the research of centers and the developmental 
efforts of labs; and the independent regionality of labs 
was in contrast to the university-based and administered 
R&D center. 10 



While the distinction between centers and labs in terms of 
their relative emphasis on basic and applied research, development, 
and dissemination appears to have been maintained in principle over 
the years, there is a continuing tension within these two sets of 
institutions as well as between them over the proper mixture of 
these activities. Indeed, the charge has been frequently made that 



10 Jerry Phillip Walker, "An Analysis of Differential 
Perceptions Toward Educational Research and Development Held By 
Professional Staff Members of Research and Development Centers and 
Development Laboratories", Unpub. Ph.D. diss. , Ohio State 
University, 1972. 
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the distinctions between centers and labs have often disappeared in 
practice if not in theory. Walker concluded that by the spring of 
1972: 

It can be seen then, that although a clear written policy 
and set of guidelines do not yet exist whereby the performance 
of laboratories and centers is delineated in terms of expected 
roles and functions, a variety of documents, events, and 
signals indicate that the expectations for the performance of 
both labs and centers have, by design or default, been 
consolidated. ^ 

Nevertheless, despite the blurring of expectations between the 
centers and the laboratories, most participants and outside 
observers seem to believe that some distinctions in emphasis do 
continue to exist between these two sets of institutions. Indeed, 
most of the outside reviews of centers and laboratories reaffirm 
the importance of that distinction. For example, the 

congressionally-mandated panel that assessed the centers and 
laboratories in 1979 concluded that: 

The Panel finds virtue in maintaining the distinction 
between the purposes of laboratories and centers. We have 
seen and find desirable the considerable overlap in their 
activities. Nevertheless we affirm the need for the two types 

11 Walker, 11 An Analysis of Differential Perceptions," p. 19. 
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of institutional capacity. Centers should direct their 

efforts within an educational problem area of national 

importance, initially determined by NIE. Through emphasis on 

basic and applied research, centers should exercise national 

leadership in their problem areas. This research could lead 

to appropriate development and dissemination activities 

related to the problem area. 

Laboratories , in contrast , should address a larger 

spectrum of educational problems. These problems will be 

defined by their regional constituents. In pursuit of their 

priorities, laboratories are likely to engage in a mix of 

applied research, development, evaluation, technical 

12 

assistance, and dissemination activities. 



In terms of the activities of the centers, perhaps one of the 
major differences of opinion among analysts is over the amount of 
time that the centers should devote to basic research. Although 
the distinction between basic and applied research is often hard to 
identify or maintain in practice, there is general agreement that 
most R&D centers do not produce much basic research. In part, this 
is because initially the Office of Education separated its 
commitments to fundamental research from its efforts to improve 

12 Panel for the Review of Laboratory and Center Operations, 

Research and Development Centers and Regional Educational 

Laboratories: Strengthening and Stabilizing a National Resource. 
Final Report, National Institute of Education (Washington, D.C. : 
U.S. Department of Education, 1979), p. iv. 
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education through research and development. As Chase observed, 
without voicing any disappointment or disapproval: 



The centers and laboratories , however , were not conceived as 

organizations devoted primarily to basic research; and few, if 

any have adequate concentrations of scientific or other 

scholarly talent to lay out research designs for fields as 

loosely specified as higher education or raising the quality 

13 

of education in a region. 

The National Research Council's analysis of fundamental 
research in 1977 agreed with Chase that relatively little basic 
research was being done by most of the centers. But rather than 
accepting this either as inevitable or desirable, they protested 
and urged the National Institute on Education (NIE) , the successor 
to the Office of Education, to emphasize the importance of 
fundamental research more in their programs. They pointed out that 
one of the four explicit goals for NIE was support for fundamental 
research. 14 

The National Research Council's more recent analysis of the 
Office of Educational Research and Improvement (OERI) reiterated 
the call for a more balanced portfolio of activities — including an 



13 Chase, "The National Program of Educational Laboratories, M 
p. 17. 

14 Sara B. Kiesler and Charles F. Turner, Fundament al Research 
and the Process of Education (Washington, DC: National Academy of 
Sciences Press, 1977). 
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expansion of support for basic research. As they put it, "the 

centers should undertake considerable more basic research than they 
15 

currently do." 

Just as there is a tension within the centers on how much 
funding should be devoted to basic, applied,, or developmental 
research, there is also disagreement over how much effort should be 
paid to the dissemination of their findings. Historically, it was 
anticipated that the labs rather than the centers would devote more 
attention to dissemination. As with many of the other overlapping 
activities between the labs and centers, that distinction in 
practice has often been eroded.* 6 

Indeed, there has been continued pressure on the centers by 
the Congress as well as OERI itself to devote more effort to 
disseminate their research findings to teachers and to the public 
as well as to other scholars. For example, in the instructions for 
applications for grants under the Educational Research and 



15 Richard C. Atkinson and Gregg B. Jackson, eds., Research 
and Education Reform: Roles for the Office of Educational Research 
and improvement (Washington, DC: National Academy Press, 1992), p. 
150. 

16 One should also note that from the very inception of the 
National Institute of Education (NIE) often there were 
disagreements over the role NIE would play in disseminating the 
results of its work to local schools, whereas Elliot Richardson, 
the Secretary of the Department of Health, Education, and Welfare, 
minimized the role of NIE in directly disseminating research 
findings to local educational institutions, some of those on the 
House Select Subcommittee on Education stressed the importance of 
NIE being more involved in such an activity. U.S. Congress, House, 
Committee on Education and Labor, To Establish A National I nstitute 
of Education: Hearings Before the Select Sub committee on Education, 
92nd Congress, 1st Session (Washington, DC: U.S. Government 
Printing Office, 1971) . 
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Development Center Program, the primacy of their research agenda 

was acknowledged. 17 But the application instructions also 

directed, in bold letters, that "Each Center Is Expected to Engage 

in an Active Dissemination Program. " Moreover, "[i]n addition to 

actively disseminating their work, Centers are expected to answer 

18 

or refer reguests from people throughout the country." While 
no one disputes the value and importance of dissemination, there is 
some guestion of what is the most effective and efficient way of 
proceeding. 

Some feel strongly that centers and researchers should 
disseminate their own materials. For example, Susan Fuhrman, 
director of the Consortium for Policy Research in Education at 
Rutgers University, cogently argues that researchers should play an 
important role in dissemination rather than just relying upon 
others. 19 Others have doubts about the recent emphasis on 



17 "OERI's primary purpose continues to be to provide the 
research base necessary to define and analyze specific educational 
problems so that teachers and other practitioners, parents, and 
policymakers can improve the guality of education." Office of 

Research, OERI, A pplication for Grants Under the Educational 

Research and Development Center Program (Washington, DC: U.S. 
Department of Education, 1990), p. 24. 

18 Office of Research, A pplicati on for Grants. 1990, p. 26. 
Christopher Cross, the Assistant Secretary for OERI, was 
particularly interested in having all of the centers increase their 
activity in dissemination. He was also a prime mover behind the 
efforts to establish the Dissemination and Knowledge Utilization 
Center . 

19 Susan H. Fuhrman, "Uniting Producers and Consumers: 
Challenges in Creating and Utilizing Educational Research and 
Development." Paper presented at the International Seminar on 
Educational Research and Development, Washington, DC, June 1-2, 
1992. 
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dissemination. Many of those who question the emphasis on 
dissemination by the R&D centers today do so because they believe 
that it comes at the expense of the quality and quantity of 
research done by the centers . If the centers were larger and 
better funded, they would not necessarily object to centers doing 
dissemination . 

One fundamental change that has occurred in the functioning of 
the centers is a significant reduction in the amount of money (in 
constant dollars) that are available for this activity while at the 
same time consuming an increasingly large proportion of the NIE (or 
OERI) total research budget. The expenditure on centers (in 
constant 1982-84 dollars) rose from $3.2 million in FY64 to $48.2 
million in FY73 — the high point in center funding (see figure 2). 
But then it plummeted to $27.1 million for FY74 and has continued 
to drop until it seems to have levelled off at $15.6 million for 
FY91. Moreover, as the overall research budget for NIE/OERI 
dropped even more dramatically than for either the centers or the 
labs, their percentage share of total resources improved 
significantly — often to the dismay of other researchers who would 
have preferred that the funds for field-initiated research had been 
preserved more. One result of this dramatic change is that at the 
same time that the R&D centers found their funding significantly 
reduced, there were even greater expectations placed on them as 
centers and the labs received most of the reduced remaining 



20 See footnote 6 for a discussion of the data sources for 
these calculations. 
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research dollars in NIE and OERI. 



Insert figure 2 



At the same time that the total funding in real dollars for 

centers was being cut drastically, decisions were made at several 

points to increase the number of centers — especially in the last 

five or six years. With 14 of the centers expiring in 1989, the 

Office of Research in OERI initiated a series of activities to 

identify what areas of inquiry seemed to be the most profitable and 

to decide how many centers should be established. Initially, OERI 

concluded that 12 centers should be established, but then added 

21 

another 7 projects for consideration. Finally, Christopher 

Cross, the Assistant Secretary of Educational Research and 
Improvement, recommended that 18 new centers be funded — thus making 
a total of 25 centers supported by OERI. 



In late 1989, the Office of Research (OR) considered three 
options — fund 12 centers, fund 5 larger centers, or fund 9 medium 
sized centers. There was strong support within OR for the funding 
of 5 centers with an annual average of $2.0 million (in current 
dollars) , but that suggestion was rejected. The five centers would 
have been ( 1 ) Student Learning ; ( 2 ) Families , Communities , and 
Young Children's Learning; (3) Middle Grades and High Schools in 
the Inner Cities; (4) Learning to Teach; and (5) Education Policies 
and Student Learning. Instead, they continued with the idea of 12 
centers, with an annual average of less than a $1.0 million, which 
then was expanded by the addition of 7 more centers fo. 
consideration . 

22 

Christopher T. Cross, "Approval Concurrence of the Centers 
to be Competed in the FY 90/91 National Educational Research and 
Development Centers Competition" (memo to the Secretary of 
Education, January 22, 1990. The final decision was option 3 which 
called for the funding of the original 12 proposed centers, the 
Dissemination and Knowledge Utilization Center, and as many of the 
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As a result, whereas there were 10-12 centers during the 
1970s, there are now 23. This means that the average amount of 
funding (in constant dollars) per center has significantly dropped 
from $2-$4 million in the early 1970s to less than $800,000 today. 
Consequently, the entire meaning of a national center changed over 
time because the current level of funding is so much less than had 
been envisioned or provided earlier. 

The National Research Council's recent examination of OERI 
bemoaned the small amount of funding for the existing centers and 
recommended a substantial increase. 

The committee cannot imagine a robust R&D center operating at 
much less than $3 million annually in core funding [current 
dollars]. Without that level of support, there will not be 
the critical mass, diversity of expertise, and scale of 
operation that are needed to tackle the difficult research 

problems and development efforts that confront the national 

23 

schools. 

Given the surprisingly small amount of total money that actually 
goes into research, as we shall later see, the National Research 

other centers as funding would allow. The Center for the Teaching 
and Learning of the Arts, which was to be co-sponsored with the 
National Endowment for the Arts (NEA) , was dropped because NEA 
decided not to support that activity. Eventually, all 18 of the 
final proposed set of 19 centers were funded by OERI. 

23 Richard C. Atkinson and Gregg B. Jackson, eds., Research 
and Education Reform , pp. 150-151. 
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Council's recommendation for the minimum size of a center actually 

may be too small to carry out fully the tasks that they envision 

24 

for each of the new centers. 

Another continuing source of change in the centers is the 
expectations of their longevity and stability. Originally the 
centers were envisioned to be institutions that were reviewed and 
renewed on a long-term basis. A "three-five plan" was instituted 
which meant that centers were evaluated in their third year to 
determine if they should be funded for another five years. 

With the establishment of the National Institute of Education 
(NIE) in 1972, support for institutions such as the centers Was 
reconsidered. A decision was made to continue centers and 
laboratories, but to introduce more competition through a "program 
purchase" policy. That is, labs and centers now had to compete 
with each other as well as with other organizations for funding for 
specific projects from NIE— a situation made all the more difficult 
by the sizable decreases in the overall NIE budget. To counter 
this move toward more competitiveness during a period of total 
budget decreases, the labs and centers banded together in an effort 
to get the Congress to earmark special funds within the NIE budget 



24 The National Research Council study did not investigate in 
any detail exactly how the funds of the existing centers are being 
utilized or just how much money remains for research once the 
overhead, the administrative costs, and the expenses of 
dissemination are paid. 



25 Mason, "Two Decades of Experience." 



for their exclusive use. 

Ronald Campbell headed a group of consultants who re-examined, 
among other things, the NIE's policies toward the centers in 1975. 
They rejected the idea of the "program purchase" for the R&D 
centers and recommended "stable funding for three to five years, at 
a level of at least $3 to $4 millior per year" (or about $5.6 to 
$7.4 million in constant 1982-84 dollars). 27 The National 
Council on Education Research (NCER) , the policy-making advisory 
group for NIE, accepted the recommendations of this report and 
began to implement them. 28 The earlier "program purchase" policy 
of NIE was abandoned. 

As part of the reauthorization of NIE in 1976, the Congress 
called for a panel of educators to review and make recommendations 
about the R&D centers and regional laboratories. The 15 members of 
the Panel unequivocally endorsed the continuation of these 
institutions: 

The Panel strongly endorses the concept of research 

26 Lee Sproull, Stephen Weiner, and David Wolf, Organizing an 

Anarchv: Belief. Bureaucracy, and Politics in the — NatlPPfrl 

Institute of Education (Chicago: University of Chicago Press, 
1978) . 

27 Ronald F. Campbell et al., R&D Fundi ng Policies of the 

National Institute of Education: Review and Recommendations . 

National Institute of Education (Washington, D.C: Department of 
Health, Education, and Welfare, 1975), p. 79. Note how much larger 
the proposed centers would be according to the Campbell groups 
compared to the recent recor emendations of the National Research 
Council. 

28 Mason, "Two Decades of Experience." 
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and development centers and regional educational 
laboratories and affirms the importance of maintaining 
and improving the stability and quality of the existing 
institutions. While finding considerable differences 
among the individual centers and laboratories as to their 

strengths and weaknesses, we find most of them now ready 

29 

for long-term relationships with NIE. 

The Panel explicitly repudiated the "program purchase" approach and 

returned to the original concept of five-year agreements that are 

3 0 

renewable after a favorable third-year review. 

In the early 1980s there were some major changes in the 
structure and functioning of NIE in regard to the centers and labs. 
First, a separate Department of Education was created and the 
Office of Educational Research and Improvement (OERI) was set up to 
oversee NIE, the National Center for Education Statistics (NCL.S) , 
the Library Programs, and several other small programs (but NIE 
remained as a semi-autonomous entity until 1985) . Center programs 
continued to be monitored by the relevant unit within NIE, but now 
each center also had a program monitor assigned to it. Moreover, 
a Laboratory and Center Coordinator was established to oversee the 



Panel for the Review of Laboratory and Center Operations, 
Research and Development Centers , pp. iii-iv. The panel 
recommended that NIE enter into long-term agreements with seven of 
the existing nine centers and seven of the eight laboratories. 

30 

Panel for the Review of Laboratory and Center Operations, 
Research and Development Center f pp. iv, vi. 
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operations of these entities within NIE. 

Second, the election of Ronald Reagan in 1980 heralded some 

major changes in the role of the centers. The ongoing effort by 

some to reverse the policy of supporting the labs and centers 

almost indefinitely (no center or lab had lost its funding from NIE 

since 1976) appeared to succeed when Congress inserted in its 

report for the Omnibus Budget Reconciliation Act of 1981 that "the 

Regional Educational Laboratories and Educational Research Centers 

shall, upon completion of existing contracts, receive future 

funding in accordance with government-wide competitive bidding 

procedures." 32 In March 1982 Edward Curran, the new Director of 

NIE, announced that NIE planned to terminate the existing five-year 

contracts for 15 of the 17 labs and centers one year ahead of 

schedule. His actions immediately provoked angry responses from 

the centers and labs as well as from other influential educators 

and some members of the U.S. Congress. While most of them did not 

challenge the principle of competition for refunding these 

institutions, they questioned the legality and wisdom of 

unilaterally terminating the funding of the existing centers and 



Mason, "Two Decades of Experience." For an analysis of the 
creation of the U.S. Department of Education, see Beryl A. Radin 
and Willis D. Hawley, The Politics of F ederal Reorganization; 
Creating the U.S. Department of Education (New York: Pergamon 
Press, 1988) . 

32 Quoted in Philip Phaedon Zodhiates , "Bureaucrats and 
Politicians: The National Institute of Education and Educational 
Research Under Reagan." Unpub. Ed. D. diss., Harvard University, 
1988, p. 81. 
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labs. 

Cur-ran 1 s attempt to terminate quickly the existing labs and 
centers failed , but the idea that such institutions would have to 
compete openly in the future succeeded • Beginning in 1985 a 
competition was set up for future labs and centers with the clear 
understanding that all of them again would have to face another 
open competition five years later (assuming that OERI support for 
their particular area of research continued) • Indeed , of the 13 
centers that were in operation during the late 1980s , 3 were 
terminated and only 5 successfully competed for continuation in 
1990 (out of 8 that had applied) • 34 Thus, centers now are seen 
as temporary 9 five-year institutions which must recompete for 
continuation every five years if OERI decides to support future 
research in that area. 

While not arguing against the principle of competition for the 
selection of centers, the National Research Council laments the 
instability created by having each center recompete frequently 
since "the 5 -year cycles are inconsistent with the need for the 

repeated iterations of research, development, demonstration, and 

35 

evaluation, which often require a decade or more." Instead, 

33 Zodhiates, "Bureaucrats and Politicians." 

34 Richard C. Atkinson and Gregg B. Jackson, eds., Research 
and Education , p. 124 . 

35 Richard C. Atkinson and Gregg B. Jackson, Research and 
E ducation Reform , p. 151. Ironically, as we shall see later in 
this report, much of the work of the centers is fragmentary and 
non-cumulative, ence the rationale for having more stable centers 
seems less persuasive under the current functioning of those 
institutions. 

3 : 
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they envision centers which might be created for 10 or 15 years and 
terminated only for inadequate performance. 



The centers generally did not compete until the 
1980s, and there is ample precedent in other federal 
agencies for negotiated renewals. One option would be to 
have competitions only every 10 or 15 years. During the 
interim period, accountability would be achieved by 
continuous monitoring and feedback, by periodic formal 
evaluations, and by basing a portion of the federal 
contribution at each 5-year renewal on the evaluations. 
A panel of researchers, developers, teachers, and 
administrators could be assembled in the fourth year to 
review internal and external evaluations of all the 
centers and judge their performance. OERI would adjust 
its contribution to each in response to the judgments; 

termination would be used only in cases of inadequate 

36 

performance. 

Certainly the National Research Council is correct in arguing 
that many educational ideas and innovations require more than a 
decade to be fully researched and developed. There are, however, 
some practical difficulties involved in not recompeting centers 
more often than every 10 or 15 years. First, if the overall number 
of centers are reduced as the National Research Council suggests, 

36 Ibid. 
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then it may be frustrating for a new administration not to be able 
to redirect fundamentally some of the major research activities in 
the centers during their four- or eight-year tenure. Second, if a 
decision is made to fund centers for 10 or 15 years, it would be 
important to have them gradually phased in so that the opportunity 
to create at least one new center would occur sooner than if all of 
the centers were established at the same time. Third, the National 
Research Councils recommendation depends, as they acknowledge, on 
the ability of OERI to closely monitor and influence the quality 
and direction of a center *s activities — something which has not 
always been true of their supervision of the existing centers «, 

Throughout the past three decades, the quality of research in 
education has often been regarded by the public and policy makers 
as inadequate and second-rate compared to that in the sciences or 
even in the other social sciences. While these observations and 
impressions may not be accurate, they have nevertheless seriously 
hurt the image of OERI and minimized support for activities such as 
the R&D centers. 37 Given this long-standing concern about the 
quality of the research sponsored by NIE/OERI, it is surprising and 
disappointing to discover that almost none of the major evaluations 
of the centers have attempted to investigate the quality of their 
research. 



For an excellent analysis of the disrespect for research in 
education historically as well as today, see Carl F. Kaestle, 
"Everybody's Been To Fourth Grade: An Oral History of Federal R&D 
in Education. H Final Report to the Committee on the Federal Role 
in Education Research, National Academy of Sciences (September 
1991) . 
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Some of the early assessments of centers did not look closely 
at the research because these institutions had just been created. 
Thus, an early congressional study of the U.S. Office of Education 
explained that "an in-depth evaluation of the work of the Centers 
was not undertaken by the subcommittee as most of the Centers are 
only 12 to 18 months old." Nevertheless, the subcommittee, on the 
basis of discussions with the Office of Education as well as 

reading some of the center evaluations, concluded that "the Centers 

3 8 

vary widely in quality." 

Most of the other major evaluations of the centers simply did 
not attempt to assess the quality of the research. For example, 
the large-scale analysis of the publications of the regional 
laboratories and research and development centers in the early 

1980s focused on a content analysis of the types of publications 

39 

produced rather than the quality of the work. 

Although the recent National Research Council analysis of OERI 
initially planned to look at the quality of the research produced, 
due to time and financial constraints they decided to omit this 
phase of their investigation altogether. 40 As a result, while 



38 U.S. Congress, House, "Study of the United States Office of 
Education," p. 224. 

39 Kendall 0. Price, "Creating and Disseminating Knowledge For 
Educational. Re form: Policy Management of the National Institute of 
Educations^ Regional Educational Laboratories and National 
Research and Development Centers." Final Report to the National 
Council on Educational Research , U.S. Department of Education 
(January 1984) . 

40 The National Research Councils proposal of work in August 
1990 specifically stated that "the study will include a close look 
at the institutions and activities supported by OERI for the 
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they conclude that "OERI has a checkered history in respect to 

quality assurance/ 1 they do not attempt to ascertain what this has 

meant in practice for the quality of research produced at the R&D 
41 

centers . 

Finally, it also should be noted that none of the evaluations 
of the centers have ever examined closely the budgets. Although 
several of the reports have commented on the diminishing amount of 
money (in constant dollars) available at the centers, they have not 
attempted to ascertain what proportion of the funds actually go to 
activities such as research or dissemination. Nor have they 
attempted to see what variations exist among the centers at any 
given time in the amount of funding for these activities. As a 
result, neither the quality of the research nor the amount of funds 
expended upon it have been analyzed or evaluated. 

II. Assessment of the Research Centers Today 

To analyze the quality of recent research produced by OERI's 
research centers, five current and two former centers were examined 
in considerable detail. The five current centers are: 

1. Center for Research on Effective 
Schooling for Disadvantaged Students (primary 



conduct of education research, particularly the labs and centers," 

41 Richard C. Atkinson and Gregg B. Jackson, Research and 
Education Reform , p. 126. 
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site at Johns Hopkins University) . 

2. Center for Research on Evaluation, 
Standards, and Student Testing (primary site 
at UCLA) . 

3. National Resource Center on Student 
Learning (primary site at the University of 
Pittsburgh) . 

4 . National Center for the Study of 
Writing and Literacy (primary site at 
University of California, Berkeley) • 

5. Policy Center of the Consortium for 

Policy Research in Education (primary site at 

42 

Rutgers University) . 



The two former centers are: 



1. National Center for Improving Science Education 
(primary site at Andover. MA.)« 

2 . Nat i ona 1 Center on Educat ion and Emp 1 oyment 
(primary site at Teachers College) . 



The seven centers were suggested by the staff of the Office of 
Research (OR) in OERI — in part because they exemplify the variety 



For convenience in discussing these centers, they will 
often be referred to by the designated name of their primary site. 
The Center for Research on Effective Schooling for Disadvantaged 
Students, for example, will sometimes be referred to simply as the 
Johns Hopkins Center «> 
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of activities in the research centers as well as because they 

display considerable variation in the type and quality of research 

produced. They are by no means intended to be representative of 

the other 20 research centers in existence today — though they 

undoubtedly reflect some of the strengths and weaknesses of those 

43 

institutions as well. As we shall later see, the five current 

centers investigated are significantly larger than those that were 

not analyzed in more detail. Therefore, while some of the specific 

assessments of these particular centers cannot and should not 

necessarily be applied to all of them, it is unlikely that a more 

extensive study of all 23 centers would have significantly altered 

the overall conclusions of this analysis. 

All of the available published^ and unpublished materials 

submitted to OERI by each of the seven centers since 1985 were 
. . 44 

scrutinized. The budgets of the centers were examined with 

particular attention to the information for FY32 for the five 



In order to be able to examine the quality of research, it 
was advisable to select current centers that had already been in 
operation before 1990 so that an ample set of research products 
were already available for inspection. Therefore, some of the new 
centers funded in 1990 or thereafter were systematically excluded 
from this more detailed scrutiny. One might object that since all 
of the centers investigated existed prior to 1990, it biases the 
analysis toward the more successful ones (which were able to win in 
the next round of competition) . The inclusion of two former 
centers which were not successful in recompeting in 1990, however, 
helps to correct for that potential bias. For a list of the 
current centers, see appendix B. 

44 

The relevant materials were assembled by the OERI center 
monitors from their own files. For the five ongoing centers, most 
of the written materials for them were produced in the five-year 
period prior to 1990. 
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45 • . 

ongoing ones. The original proposals for the centers were read 

as well as the comments of the reviewers who considered them for 
46 

funding. In addition, for most of the centers, the third-year 

renewal applications and the comments of their reviewers were 

examined. Moreover, the written comments of the 0ERI center 

monitors and the responses of the centers to these suggestions were 

analyzed. All five of the ongoing centers were visited which 

provided an invaluable opportunity to discuss their research work 

47 

as well as their other activities. Each of the OERI center 

monitors for these seven institutions were interviewed (most of 

them more than once) . Copies of an earlier version of this essay 

were distributed within OERI for comments and an OR staff seminar 

was devoted to a discussion of the contents of this report. A 

revised version of this report was sent to both the center and 

laboratory directors for comments. The Mary Kennedy, chair of the 

Organization of Research Centers (ORC) , provided extensive comments 

48 

on the draft report. 



45 

The special attention to the FY92 budgets was because they 
contained more detailed information about the budgets of the 
individual research projects. For the period 1985-1990 most of the 
centers did not provide detailed budget breakdowns for their 
individual research projects. 

46 .... 

For most of these centers, the 1985 original application as 

well as the 1988 renewal packets were examined. In addition, for 
many of them the 1990 grant application was also examined. 

47 . . 

The Berkeley Center was visited on June 5-6, 1992, the UCLA 

Center on June 7-8 , the Johns Hopkins Center on June 17 , the 
Rutgers Center on July 13, and the* Pittsburgh Center on August 20. 

48 ... 

A copy of her letter as well as of my specific reactions to 

her criticisms can be found in appendix C. 
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A. Distribution of Research Activities of Centers 



As we have already discussed , the centers have greatly 
diminished in size over time. In constant dollars, the funding for 
the average center today is about one third that of its counterpart 
more than twenty-five years ago. Today , most of them are no longer 
the large-scale national institutions that were envisioned 
initially. In FY92 there were 23 centers with a total funding of 
$26.8 million (in current dollars). Or, the average center was 
funded at an annual rate of about $1.2 million. Only five of the 
23 centers received at least $1.5 million and four of them were 
funded at less than a $1 million. The Center on Assessment, 
Evaluation, and Testing (UCLA) received the most support from OERI- 
-$2.7 million. 

While an annual rate of funding of about $1.2 million may 
appear to be a large amount of money, it is important to realize 
that not all of this money got s directly to support research. 
Instead, a substantial portion of the funds are allocated for other 
expenditures such as overhead, administration, and dissemination. 
Therefore, in order to examine the amount and distribution of funds 
expended for research, a more indepth analysis of the FY92 budgets 
of the five current centers was undertaken. 

The centers at Berkeley, Johns Hopkins, Pittsburgh, Rutgers, 
and UCLA were funded at a higher level than the others. The 
average annual funding in FY92 for these centers was $1.7 million 
while the comparable figure for the remaining 18 centers was only 
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49 



$1.0 million. Four of the five centers investigated received 
at least $1.5 million and the other one got $1.0 million. 50 

Indirect costs are given to research institutions to defray 
such expenses as maintaining buildings and providing administrative 
assistance. These indirect costs vary from one university to 
another and are usually a set, negotiated percentage of the direct 
costs of a project. There is also considerable variation among 
institutions on just how much, if any, of the indirect costs are 
given back directly to the unit doing the actual research. For 
these five centers, the indirect costs represent a substantial 
expenditure. Of the $8.5 million provided by OERI to these centers 
in FY92, approximately $2.35 million went for indirect costs — or 
27.6 percent of the total funding. 5 ]" The proportion of money 



49 

The information for the more detailed examination of the 
FY92 budgets came from the Application for Continuation, Part I and 
Part II. The data were analyzed by hand because the computerized 
OERI Project Management Information System 1 s (PMIS) did not provide 
the type of categories about research expenditures that were 
necessary for this analysis. 

50 . 

Thus, four of the five largest centers were used in this 
analysis — including the UCLA center which received the $2.7 
million. It is important to note, however, that almost all of 
these five centers were considerably smaller in the period FY86 to 
FY90. The average size of these five centers in FY90 was $1.14 
million (in current dollars) and $1.68 million in FY92 — a 47 
percent increase. The UCLA Center in particular experienced the 
largest increase in funding, from $1.0 million in FY90 to $2.7 
million in FY92. As a result, while these five centers are 
relatively large in FY92 compared to the others, prior to FY91 they 
were much closer to them in size and therefore the products that 
are available from that earlier period are more typical and 
representative of the other centers at that time. 

51 

The 27.6 percent is not the rate of indirect costs, but the 
proportion of the total funds that went to this item. If one 
calculates the rate of overhead based upon the overall direct costs 
of these five centers, that figure is 38.1 percent. Since any 
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spent on indirect costs varies considerable among the centers — from 
about 22 percent to 36 percent of the total funds (see figure 3). 



Insert figure 3 



Administrative expenses and the costs of disseminating their 

findings are major expenditures for most centers. As we noted 

earlier, OERI has often emphasized the importance of dissemination 

and expects centers to play an important role in this process. 

Since some of the centers in their budget reports did not always 

clearly differentiate between the costs of administration and 

dissemination, it was necessary to group these expenses together 
52 

here. Overall, 19.6 percent of the funds for these five 



given center often has several colleges and universities that 
participate in its activities, the indirect rate for each of them 
can vary substantially. One of the difficulties with OERI's 
current computerized PMIS is that it does not always allow the 
analyst to distinguish the indirect costs from the direct ones. 
For example, in PMIS printouts one can distinguish the direct and 
indirect costs for the primary site institution, but not for the 
subcontractors. Although this information is usually available in 
the original budget application forms, the current computerized 
PMIS does not make that distinction. Another problem is that 
occasionally the centers failed to provide detailed information on 
the budgets of the subcontractors. In these few instances it was 
necessary to contact the center to obtain that necessary 
information. 

52 

Most centers did provide data on the costs of 
administration and dissemination, but some did not distinguish 
between the two (and sometimes, of course, it might be difficult to 
entirely separate these out since some of the project 
administrators were also very active in disseminating the results 
of their center's research). Moreover, since some of the research 
projects also include money for dissemination, it was necessary to 
estimate that amount using information from the distribution of 
activities as reported by the centers on their Project Input Forms 
for their continuation budgets. Naturally, these activities 
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centers went for administration and dissemination. There was even 

more variation in their expenditures for administration and 

dissemination than for their indirect costs (see figure 3). 

Whereas the Johns Hopkins Center spent only 5.3 percent of its 

total funds for these activities, the Berkeley Center spent 51.3 
53 

percent . 

One of the interesting, but difficult policy issues is how 

should OERI interpret the indirect and administrative costs of the 

centers. The indirect rate for each university has already been 

established through negotiations and covers the overhead expenses 

of maintaining research activities — including some funds for 

general administrative purposes. Therefore, how much additional 

funding is needed to cover the expenses of running the centers? 

And should centers such as the one at Johns Hopkins be expected to 

charge less for administrative costs because their relative rate 

for indirect costs is considerably higher than that of some of the 

54 

other institutions like the Berkeley Center? 



distributions are only approximations and therefore the final 
figures for dissemination are only estimates. 

53 It should be noted that the Johns Hopkins Center did devote 
a lot of effort to dissemination, but they did not charge OERI for 
that activity. Therefore, the variation in the amount of money 
spent does not always reflect the actual amount of effort made 
since the source of the funding for these activities might vary 
from one center to another. For example, some of the 
administrative costs in one center might be paid by the indirect 
costs while comparable services at another center might be charged 
directly to OERI. 

54 In fact, the Johns Hopkins Center does not charge OERI very 
much for administering the center compared to some of the other 
centers which have a lower indirect cost rate. 

One should also note that some of the centers work very hard 
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The net result of the expenditures for overhead, 
administration, and dissemination is that the amount of funds 
available for research is considerably smaller. Overall, only 52.9 
percent of the total money for these five centers was devoted to 
research. Again, there was great variation among the centers in 
how much they spent on research (see figure 3) . Whereas the 
Berkeley Center spent only 26.0 percent of its overall funds for 
research projects, the Pittsburgh Center spent 62.5 percent. Thus, 
whereas the average annual grant for these five centers was $1.7 
million, the average amount actually spent on research was only 
$890,991 (and ranged from $2t'0,262 at the Berkeley Center to 
$1,501,441 at the UCLA Center). 55 



to reduce the overall amount of indirect costs in order to have 
more money for other activities. Sometimes this is done by 
categorizing research expenses such as supplies under 
administrative costs (if administrative costs are not subject to 
indirect costs). Other centers try to have their facilities off- 
campus which have lower indirect cost rates. Reducing the indirect 
costs can have a very large impact on the amount of money available 
for research, but centers are reluctant to discuss publicly their 
tactics to reduce those expenses because this might upset 
university officials who would like to have more indirect costs. 

55 The figures provided only indicate the amount of OERI funds 
spent on research. Some of the centers supplement their OERI 
monies with cost-sharing contributions of salaries, services, or 
space. For example, the Rutgers Center contributes an additional 
$123,530 for their research projects — about a one-sixth increase 
over their OERI-funded portion of the research budget. 

The overall budget of soite of the centers is also supplemented 
by outside funding. The direct research funds at the Rutgers 
Center rises from $359,413 to $2,156,327 (including over $100,000 
from another one of the OERI-funded centers). Interestingly, the 
proportion of NIE/OERI funds for the Pittsburgh Center has declined 
considerably over time ( 1975—82 . 6 percent ; 1984 — 56 . 3 percent ? 
1988 — 29.7 percent; 1993 — 12.8 percent). Most centers, however, 
appear to be much more heavily dependent upon OERI funds than 
either the Rutgers or Pittsburgh Centers. 
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Given the rather modest amount of money devoted to research at 
the five centers, they had the choice of concentrating it on a few 
large projects or spreading the money among several smaller ones. 
Although the definition of an individual research project may not 
be identical among the centers, the analysis of these particular 
institutions suggests that most centers chose to fund multiple 
research projects. The number of separate research projects for 
these five centers ranged from 9 to 25 (most of them reported about 
10 research projects) — and some of these were even further 
subdivided in practice. 56 As a result, the total annual amount 
of money directly available for any given project was rather modest 
and in effect usually precluded any large-scale research 
undertakings. While only 30 percent of the reported individual 
research projects in FY92 received more than a $100,000 in direct 
costs, almost half of them were funded at less than $50,000 a 
year. 57 Moreover, given the substantial salary costs for many of 



The Johns Hopkins Center listed 25 individual research 
projects, but some of these were small subprojects of a larger 
effort. If one compares the Johns Hopkins Center research projects 
to those of other centers, the actual number of different projects 
probably would be rather similar. For example, while the UCLA 
Center lists only a few larger projects, in practice many of them 
are composed of several smaller ones. 

57 Naturally, if centers use cost-sharing or outside funds to 
supplement their OERI-funded research projects, the distributions 
reported here would exaggerate the number of small research 
projects. However, most of the centers do not appear to use their 
outside funding to supplement their individual OERI-funded projects 
(though the new projects created may complement the work being done 
on an OERI-funded project) . Moreover, while cost-sharing certainly 
may enhance the size of some of the smaller research projects, its 
overall effect is still probably unlikely to change the general 
picture very much. In any case, it certainly w aid be useful to 
undertake a much more indepth examination of how money is spent on 
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the academics and professionals (including their fringe benefits), 
it meant that most researchers were paid only to devote a very 
small fraction of their time to these projects — often just one or 
two months during the summer. 58 Although the pressures to act as 
a national research center contributed to the desire to provide 
broad coverage through multiple projects , the limited funding meant 
that most of the research funded by the centers could not really 
address systematically many of the larger questions in the field. 

B. Research Topics of Centers 

There has always been considerable variation in the types of 
research that centers have done. As pointed out earlier , there has 
been a long and continuing tension between doing basic and applied 
or developmental research in the centers. Therefore, it is useful 
to look at the distribution of the types of research by the five 
centers. According to the Project Input Forms for FY92, research 
was subdivided into seven categories: (1) policy studies, (2) 
evaluation, (3) basic research, (4) applied research, (5) 
development, (6) statistics, and (7) dissemination. Naturally, 

individual research projects-- especially since the definition of an 
individual research project is not always consistently or clearly 
specified by the centers on the PMIS forms. 

The recent National Research Council study of OERI 
discovered that the principal investigators of the research spend 
only an average of one-fourth time on their center-funded work. 
Richard C. Atkinson and Gregg B. Jackson, Research and Education 
Reform , p. 65. Undoubtedly many of the academics and professionals 
involved in these research projects devoted more time to them than 
they were paid by the OERI grant. 
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there is considerable overlap among these categories and centers 
appear to vary amongst themselves on how to classify similar types 
of research. 59 Since the dissemination activities were already 
considered earlier, this analysis will confine itself to the first 
six categories. 

Given the nature of the categories and how this information 
was assembled, it does not make much sense to analyze the results 
too closely. It is clear that among these five centers , very 
little attention is being given just to assembling statistics (see 
figure 4) . Almost half, of the research projects were designated as 
applied (43.3 percent) and evaluation and development were closely 
split (11.3 percent and 13.5 percent respectively). Among the five 
centers, there were large differences in their distribution of 
research in these categories. For example, the Rutgers Center for 



The information on the types of research activities comes 
from the Project Input Forms for FY92. For any particular research 
project, it was expected that those who filled out the forms could 
split the allocation of the budget activities among the seven 
categories rather than just having to characterize them as one form 
or another. As mentioned previously, these forms were filled out 
by personnel at the centers and reflects their judgement on how 
much of the research effort was devoted to various activities. In 
some cases, the center individual who filled out the forms did not 
show their estimates to the researchers themselves. When this was 
pointed out at my site visits to the centers, they were given the 
opportunity to redo any of their previous estimates. Only the 
Rutgers Center provided new estimates and they did a major revision 
which included moving "some of the communications, supplies and 
clerical dollars from dissemination and institutional activities. 11 
Susan Fuhrman, "Letter to Maris Vinovskis," (August 14, 1992) . The 
figures used here are the revised estimates they have provided. 
Naturally, these are only crude estimates and need to be treated as 
such. Moreover, the definitions of each category may vary from one 
project to the next as well as from one center to another. 
Nevertheless, they may provide a rough and useful first 
approximation of the distribution of research activities in the 
centers. 
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Policy Research in Education, not surprisingly, devotes 38.0 
percent of its research effort to policy analysis while the 
Berkeley Center for the Study of Writing and Literacy classifies 
76.5 percent of its research work as applied. 



Insert figure 4 



Interestingly, there is very little effort devoted to basic 

research (see figures 4 and 5). Among the five centers, only 18.7 

percent of their research is basic — ranging from 4.0 percent at the 

UCLA Center to 37.7 percent at the Rutgers Center. While the 

distinction between basic and applied research, for example, is not 

always clear cut, there does not appear to be much doubt that these 

centers do not provide much basic research — a criticism which was 

made in the recent National Academy study of OERI. 60 Of the $8.5 

million in total funds allocated to these five centers, $841,103 of 

6 1 

it went for basic research. While one might argue that basic 
research is better done in settings (such as field-initiated 



Atkinson and Jackson, eds, Research and Education Reform . 

61 The definition of what is H basic M research is not always 
clearly or consistently used by OERI and the centers. For example, 
the Rutgers Center initially categorized 8.3 percent of its 
projects as basic research; however, after consulting the principal 
investigators of the individual research projects as well as 
rethinking their answers to the PMIS forms, they categorized 37.7 
percent of their projects as involving basic research (much of this 
change came from shifting activities from policy studies and 
evaluation to basic research) . Given the volatility in describing 
projects, perhaps it may be useful for OERI and the centers to 
devote more attention to defining the PMIS more clearly and 
consistently. 
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research) other than the centers, the fact that so much of OERI f s 

research funds are allocated to the centers and labs means that 

62 

relatively little basic research is currently being funded. 



Insert figure 5 



It is also interesting to observe that development is 
relatively neglected at these five centers as well (see figure 4) . 
Only 13.5 percent of their research dollars was expended on 
development — even though the formal title of the centers as 
"Research and Development Centers" suggests a higher priority for 
development than it appears to be receiving. The relative lack of 
attention to development in the centers is illustrated by the fact 
that recently some of the Office of Research (OR) staff were 
uncertain whether or not the word "Development" actually appears in 
the formal designations of the centers. Moreover, several 
individuals at the centers suggested that the word "development" 
should be dropped from the formal title since they did not feel 



The terms "basic" and "applied" research are not 
necessarily the best way to characterize the type of research done 
at the centers. Perhaps we should borrow from the Social Science 
Research Council (SSRC) the use of the phrase "mission-oriented 
basic research." They define mission-oriented research as 
"research in which practical concerns guide scientists 1 choice of 
topics. The research is conducted, however, in ways that do not 
necessarily yield immediate or directly foreseeable applications." 
David L. Featherman, "Mission-Oriented Basic Research," ifceia&jL 
social Science Research Council , 45, 4 (December 1991), 75. Most 
of the work done at the centers under the rubric of basic research 
would be more accurately described as mission-oriented basic 
research since each of the centers has a clearly defined mission. 
Interestingly, the concept of mission-oriented research was 
employed by scholars in education in the early 1970s. 
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that they were expected by OERI to place much priority in this 



63 

area. 



But the concept and practice of development has not died out 

altogether at the centers* For example, the Johns Hopkins Center 

had previously developed cooperative learning as a teaching 

64 

strategy and now employ it in their Success for All Program, 

Similarly, the Pittsburgh Center has been developing its Math 3 

Program (Making Mathematical Meaning) which draws upon their long- 

. . 65 

term, pioneering work in the area of cognition. 

The range of topics covered by the current centers is very 
large (see appendix B for a list of the centers) • Indeed, one of 
the reasons given for expanding the number of centers was to be 
able to provide research on more issues than might have been done 

with fewer centers. Almost any imaginable educational issue or 

66 

problem could be placed under one of the existing 23 centers. 
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The decreasing importance of development in general at 
NIE/OERI is discussed in greater detail in the analysis of the 
quality of research and development at the OERI-funded regional 
educational laboratories. 

64 Stephen Balkcom, "Cooperative Learning," Education Research 
Consumer Guide . 1 (June 1992). 

65 While the Pittsburgh Center has done important and 
innovative basic research in the area of math and cognitive 
structures, it has taken the center more than a decade to move into 
the developmental phase of this undertaking. Perhaps we need to 
explore wa/s of speeding up the process from basic researrZi to the 
development of more effective teaching practices. 

66 While almost anything might be investigated in one of the 
centers, in fact there are several implicit assumptions in the 
overall approaches of the centers. For example, most educational 
problems and topics are approached from a current perspective with 
relatively little attention to any historical considerations. 
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Ivor Pritchard, Director of the Education and Society Division 
in 0R / argued that the proposed set of centers in 1990 were based 
upon a fragmented and incoherent view of educational development 
and needs: 

This research agenda is also fragmentary and diffused • This 
research agenda does not identify a coherent, compelling 
vision of education and what education research can contribute 
to its improvement . Despite the number of centers , the 
proposed missions do not cover the full range of standard 
education topics, and are not related to one another in any 
systematic way. If the centers could be connected to a 
coherent vision of education, the impression created by the 
number of centers would be modified by their place in a single 
framework. But there is no such framework evident in the 
proposed centers, which gives them a random, disorganized 
quality. 

Another characteristic of many of the centers is that they 
define and describe their research agendas very broadly, but in 
fact the actual research they produce is much more limited and 
focused. The disparity between the broad project titles and 
mission statements and the narrower research efforts is not unique 
to the centers; this is very common in the social sciences in 

67 Ivor Pritchard, "Memorandum to Milton Goldberg, Director, 
Office of Research," Office of Educational Research and Improvement 
(January 2, 1990) . 
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general. Lacking in many of tba center descriptions and 
discussions is a frank acknowledgement of the limited 
generalizations that one will be able to make on the basis of the 
specific research projects undertaken. It is important to note 
this distinction because anyone who just reads the project titles 
and broad project descriptions will over-estimate just how many of 
our educational problems are being effectively and systematically 
addressed by the research done in the centers. 

If the amount of research done in these five relatively large 
centers seems limited, it is . sometimes entirely missing in the 
smaller, three-year centers that were funded annually for about 
$500,000. For example, the National Center for Improving Science 
Education explicitly renounced doing any research in their 
successful grant application in 1987: 

The nation does not need yet another entity to conduct 
research on the processes of teaching and learning science, 
nor can such research be done adequately for the amount 
allocated to OERI's science content center. What is needed is 
an organization committed to assimilating the results of 
today 1 s fervent activity, and creating bridges without which 
no coherent, integrated contribution will be made to science 
teaching and learning. 68 



Network, H A Proposal for a National Center for the 
Improvement of Science Teaching and Learning. 11 Grant proposal 
submitted to OERI (July 10, 1987), abstract, p. 3. 
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Thus, while OERI believed that small centers should and could do 
useful original research, at least one of them candidly admitted 
that this was neither feasible nor desirable • 

Although the center project titles and mission statements 
often seem to present a coherent and coordinated conception of 
their activities, in practice the projects sometimes appear to be 
more of a collection of loosely-related research undertakings. 
Some centers appear to lack a coherent and detailed research agenda 
as opposed to providing a broad framework and funding for the 
individual interests of .the faculty. Although one can obviously 
value and appreciate the diversity of research activities supported 
by the centers, given the limited funding available perhaps a more 
focused approach might be more desirable. 

For some activities, there may be an implicit overarching 

theory or plan, but it often is not evident from the individual 

70 

publications of a center. Therefore, someone reading the 

products of a particular center without having access to their 
grant application or a personal discussion with one of the 



Given the limited funding, the recent National Research 
Council study also questioned the wisdom of spreading the research 
activities of each center across several universities* Richard C. 
Atkinson and Gregg B. Jackson, eds., Research and Education Reform, 
p. 65. 



The various works on assessing history at the UCLA Center 
may seem somewhat disconnected and sporadic to the outside reader 
because the broader underlying research strategy for this area has 
not been explicitly stated. However, in my visit to the UCLA 
Center, that larger vision became more evident in talking with Eva 
Baker, one of the Centers co-directors. Moreover, in the 
subsequent continuation application, the UCLA Center clarified its 
overall themes and visions. 
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researchers might not see the coherence or relevance of the entire 

undertaking. Quite frequently there are also useful individual 

pieces of research that do not appear to be an integral part of an 

71 

ongoing research strategy. Moreover, in some centers one gets 
the impression that certain portions of the research are more part 
of an overall, ongoing systematic research effort than other 
segments • 

Sometimes otherwise well-designed and thoughtful research 
projects are not followed up adequately to provide the needed 
information for educators and policy makers. The Rutgers Center 
studied differential treatment of school districts by states. They 
choose four states (Kentucky, New Jersey, South Carolina, and 
Washington) which were either taking over control of deficient 

school districts or easing the state education regulations for 

72 

those that were exemplarary in their performance. 
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The very interesting critical analysis of studies of 
cultural literacy by the UCLA Center appears to be more of an 
isolated project than part of their broader research strategy. In 
other words, after one reads the piece and asks what is the next 
logical follow-up to that study, one discovers that none has been 
planned. 

72 For an analysis of these four case studies as well as much 
more detailed information about; each one of them, see Susan H. 
Fuhrman and Richard F. Elmore, "Takeover and Deregulation: Working 
Models of New State and Local Regulatory Relationships, M Consortium 
for Policy Research in Education, RR-024 (April 1992); Patricia 
Fry, Susan H. Fuhrman, and Richard F. Elmore, "Kentucky's Program 
for Educationally Deficient School Districts: A Case Study, " 
Consortium for Policy Research in Education, TC-005 (April 1992); 
Margaret Dolan, "State Takeover of a Local District in New Jersey: 
A Case Study," Consortium for Policy Research in Education, TC-008 
(April 1992); Susan H. Fuhrman, Patricia Fry, and Richard F. 
Elmore, "South Carolina's Flexibility Through Deregulation Program: 
A Case Study," Consortium for Policy Research in Education, TC-007 
(April 1992); Patricia Fry, Susan H. Fuhrman, and Richard F. 
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Unfortunately, they were not planning to follow up on the effects 

of these takeover actions on the school districts — thus denying 

themselves the opportunity to really assess the effects of these 

73 

different state policies on education practices and outcomes. 

Often there is even a lack of communication and coordination among 

74 

individual researchers within a center. 
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Elmore, "Schools for the 21st Century Program in Washington States 
A Case Study," Consortium for Policy Research in Education, TC-006 
(April 1992) . 

The reports of the four case studies themselves are basically 
descriptive and deliberately non-analytic. The justification for 
not providing more analysis within each of the case studies is that 
they are intended to be used for teaching purposes and the 
author (s) wants students to do their own analyses and draw their 
conclusions. Unfortunately, since the four case studies do not 
include more detailed information about the conflicting views of 
the participants in an appendix, the usefulness of these case 
studies for teaching purposes is limited. Moreover, the Rutgers 
Center had done a reanalysis of the data used by South Carolina to 
decide which schools to deregulate, but did not provide any of 
those data in the case study or even cite that report in that 
publication. 

73 

Despite the absence of much information about the 
educational impact of the takeovers, the authors appear to draw 
some rather strong conclusions about the value of takeovers. 
Fuhrman and Elmore, "Takeover and Deregulation," pp. 27-28. 
Indeed, one can even find hints in the two case studies that they 
used of some positive, short-term educational benefits as a result 
of those takeovers. Fry, Fuhrman, and Elmore, "Kentucky's 
Program"; Dolan, "State Takeover of a Local District." Given their 
decision not to follow up the educational impact of these 
takeovers, perhaps the authors should have been more circumspect 
about their conclusions of the lack of any educational benefits due 
to the takeovers. 

74 For example, while the Pittsburgh Center has several 
projects that involve the analysis of teaching history, they are 
not coordinated amongst themselves. In pari: this is because the 
individual researchers are* from different disciplines and only use 
history as a topic for their analyses. Given the sizable 
investment in history at the Pittsburgh Center, the activities of 
these individual researchers should be more coordinated so that 
they can build upon each others efforts whenever possible. 
Moreover, researchers should make more of an effort to involve 

6 j 
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Given the large numbejr of R&D centers funded and the broad 

mandate that each of them enjoys, it is not surprising that there 

is some overlap in their activities. The Berkeley, UCLA, and 

Pittsburgh Centers are evaluating the teaching of history in 

schools; UCLA and Berkeley are both assessing writing; and several 

centers are concerned with the diffusion and implementation of 

educational reforms at the state level. Although OER1 continues to 

try to improve the communication among the centers, one has the 

impression that sometimes closely-related research projects in 

75 

different centers have little interaction with each other. 

One of the first activities that a researcher undertakes in a 
new project is a synthesis and critical review of the existing 
scholarly literature in order to ascertain what needs to be done. 
In field-initiated grants this synthesis is usually done as part of 
the normal grant application process since such information is 
considered essential to convince the outside reviewers of the 
feasibility and desirability of any proposal. Many center grant 
applications, however, do not provide a thorough, critical review 
of the existing literature for their proposed research activities — 

disciplinary experts in their projects. Thus, the study of 
representations of the American Revolutionary era might have 
profited from the involvement of an academic expert on the American 
Revolution who would be more familiar with the recent scholarly 
work in this area. Isabel L. Beck, Margaret G. McKeown, and Gale 
M. Sinatra, "The Representations That Fifth Graders Develop About 
the American Revolutionary Period from Reading Social Studies 
Textbooks, " Learning Research and Development Center, University of 
Pittsburgh (December 1989) . 

75 There has been relatively little interaction and co- 
ordination, for example, between the history projects at Berkeley 
and UCLA. 



54 

perhaps in part because they sometimes unfortunately devote more 
energy to perfecting their mission statements and to describing 
broadly their research plans and dissemination activities rather 
than presenting a detailed research design. Moreover , the total 
page limit on applications discourages any lengthy literature 
reviews. Therefore , many of the center research projects undertake 
a synthesis of the existing literature as their initial task. 

There is nothing wrong with synthesizing the existing 
literature. Indeed , as mentioned above, it is an essential step in 
the research process and needs to be done — especially as new 
perspectives are introduced and considered. Yet sometimes the 
production of review essays of the existing literature appears to 
be a large, if not the major, part of a particular project rather 
than just a small, but useful portion of the entire research 
process. Moreover, centers sometimes delay developing and 
providing detailed research designs of some of their work while 
they await the syntheses of the existing research. As a result, 
the actual research in a given area may be significantly delayed in 
anticipation of these reviews. While most of these reviews are 
useful and helpful, some of them also seem to be only slight 
improvements or expansions of existing reviews of the literature. 
Consequently, many of the reviews of the literature are never 



For a useful review of the value of synthesizing literature 
and the difficulties of implementing those findings in practice, 
see Naida C. Tushnet, "Synthesis and Translation: Will It Be Easier 
For Users To Discover Meaning, Truth, and Utility in Research?" 
Paper presented at the AERA Annual Meeting, San Francisco, April 
1992. 
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published — perhaps because their utility to a broader audience is 
limited or because they duplicate previously published materials. 

Another problem with some of the centers is that they 
sometimes seem to be too eager to produce and distribute materials 
of limited interest and utility* The Berkeley Center, for example, 
has produced nearly 150 articles and essays in a seven-year period- 
-some of which appear to be too preliminary to be of much use to 
others* Similarly, while the UCLA Center has generated some 
excellent papers, it has also printed a set of overheads from an 
assessment workshop that perhaps could have been more effectively 
and efficiently distributed as a xerox copy rather than as part of 
their official deliverables* 77 As a result, not only are scarce 
resources for dissemination being expended on materials of limited 
utility, but the unevenness of the quality of center products does 
not do much to enhance the reputation of the institutions involved 
and raises questions about their publication standards and review 
process • 



C. Quality of Research 



As mentioned earlier, there has been considerable discussion, 
but almost no analysis, of the quality of research produced by the 
centers. Based upon a fairly extensive and systematic reading of 
the materials produced at seven of the recent centers as well as a 

77 Joan Herman, "High Quality Performance-Based Assessment," 
Center for the Study of Evaluation, Standards, and Student Testing 
(November 1991) • 
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more cursory glance at some of the products from a few of the other 
centers, it appears that the quality of the research produced is 
mixed—both among centers and within centers. Some of the research 
produced is of very high social science quality while other work 
could be substantially improved and expanded. 

The National Center on Education and Employment is an example 
of a recently closed center that produced some high quality 
work. 78 For example, their study of youth training in the United 
States, Britain, and Australia was a sophisticated statistical 
analysis of several national longitudxnal surveys of young men. 
They discovered, for example, that youth in the United States 
initially received less training upon entry into the labor force 
than In the other two countries, but that over time they catch up 
with their counterparts elsewhere. However, better-educated youth 
in the United States were much more likely to receive that 

additional training than others even though the more disadvantaged 

79 

youth might have benefited the most from that assistance. 

Similarly, their series of works about the changing nature of 



The Center commissioned me to do a paper on the economy 
and education in nineteenth-century American and invited me to 
attend a national conference they sponsored. Maris A. Vinovskis, 
"The Role of Education in the Economic Transformation of Nineteenth 
Century America." National Center on Education and Employment, 
Conference Paper No. 9 (December 1989) . While my temporary and 
rather minor involvement in their efforts could potentially 
introduce a bias in my reading of the rest of their work, I have 
tried to remain as objective as possible and have used the same 
standards for assessment as I have applied to the work from other 
centers . 

79 Hong Tan, Bruce Chapman, Christine Peterson, and Alison 
Booth, Youth Training in the United States, Britain, and Australia 
(Santa Monica, CA: RAND, 1991) . 
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the American economy and how it may impact on the skills needed by 

future workers was generally carefully and thoughtfully done. Some 

of the projects were extensive , critical reviews of the literature 

on schools and jobs while others were technical analyses of the 

8 0 

relationship between education and the economy. One of the 

strengths of the center was its ability to attract distinguished 

and accomplished economists and persuade them to turn their 

attention and analytic skills to the issues of schooling and the 
81 

economy • 

The Center for Research . on Effective Schooling for 
Disadvantaged Students at Johns Hopkins University is another 
example of an institution which has done some important and 
innovative research. Particularly impressive is their efforts in 
early education interventions to help at-risk children through 
their "Success for All" Program. By working closely with all 
students in their first years in school , the "Success for All" 



Gary Natriello, "The Role of the School in Preparing 
Students for Work." Final Report to OERI (June 1987); Joseph 
Altonji, "Controlling for Personal Characteristics , School and 
Community Characteristics, and High School Curriculum in Estimating 
the Return to Education. " National Center on Education and 
Employment, Technical Papers, No. 14 (July 1990) . 

81 For example, Jacob Mincer did a series of studies for them. 
Jacob Mincer, "Human Capital Responses to Technological Change in 
the Labor Market." National Center on Education and Employment, 
Technical Papers, No. 9 (November 1989) ; Jacob Mincer, "Job 
Training, Wage Growth, and Labor Turnover." National Center on 
Education and Employment, Technical Papers, No. 19 (November 1990) . 

It is also interesting 1 1 observe that this Center initially 
experienced considerable difficulty in getting itself established 
and delivering its products on a timely basis. However, they were 
able to make significant improvements in their organization and 
activities and produce some first-rate reports. 
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Program is able to help most of these students who are enrolled in 

some of the more disadvantaged areas of the inner cities. Their 

research shows that a much higher percentage of the students 

enrolled at one of their programs are able to read than their 

82 

counterparts in the control schools. Interestingly , they found 

that intensive tutoring of students in the first grade, for 

example, may be more effective in preparing them to read than 

83 

having been enrolled in a Head Start Program. 



While initially the "Success for All" program appears to 
have hoped that it would improve the reading skills of everyone, it 
now appears that a certain proportion of the most disadvantaged at- 
risk children still do not meet the minimal standards of success in 
school. As they put it, "Will Success for All ultimately bring all 
children to grade level reading? Given the program's commitment to 
avoiding retention and special education and to the fact that most 
Success for All Schools" are in very disadvantaged neighborhoods, it 
seems unlikely that the program will truly ensure grade-level 
performance for every single child. However, the program does 
substantially reduce the number of children performing below level, 
and this effect is increasing with each successive year of 
implementation." Center for Research on Effective Schooling for 
Disadvantaged Students, "Research on Success for All," Unpublished 
manuscript, chapter 7 (1992), p. 23. 

83 

Much has been written about the efficacy of Head Start, but 
most researchers now acknowledge that the results of most 
systematic evaluations of those programs are either inconclusive or 
problematic. Ron Haskins, "Beyond Metaphor: The Efficacy of Early 
Childhood Education," American Psychologist 44, 2 (February 1989): 
274-82. Moreover, most of these evaluations have not taken into 
consideration whether the money expended on Head Start might be 
used more effectively for other educational efforts. Therefore, it 
is especially interesting that Robert Slavin and his colleagues 
found that among nine different interventions they examined, 
tutoring proved to be the most effective — even compared to 
participation in a Head Start program. Robert E. Slavin, Nancy L. 
Karweit, and Barbara A. Wasik, "Preventing Early School Failure: 
What Works?" Center for Research on Effective Schooling for 
Disadvantaged Students, Report No. 26 (November 1991) . For a 
broader discussion of our changing attitudes toward early 
education, see Maris A. Vinovskis, "Early Childhood Education: Then 
and Now," Daedalus , 122, No. 1 (Winter 1993), 151-176. 
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Some of the centers have done important analyses of federal , 

state, and local policies • For example, the Rutgers University 

Center sponsored and assembled a very timely and useful collection 

of essays on systemic reform • The essays explore the theoretical 

and practical aspects of a more coherent reform strategy and 

present the differing views of education policy analysts. The work 

of the Rutgers University Center is being widely used by education 

policy makers in Washington, DC and this particular volume serves 

as a thoughtful, but complex introduction to the notion of systemic 
84 

reform. In general, the quality of the policy studies of the 
Rutgers University Center tends to be better than most of the 
comparable policy work at the regional educational laboratories. 

These are only a few examples which are intended to 
demonstrate that many of the existing centers are doing first-rate 
social science research on education. Therefore, critics who claim 
that the quality of research produced by these centers is uniformly 
low and that their results are unsound and unreliable would do well 
to look more closely at the work of some of these centers. 

If some centers and some projects within centers are doing 
important and scientifically sound research, others still have some 
room for improvement. Often centers which do first-rate work in 
one or more areas, are not equally capable in others. For example, 
the National Center on Student Learning at the University of 
Pittsburgh is certainly one of the stronger research centers funded 



Susan H. Fuhrman, ed. , Designing Coherent Education Policy 
(San Francisco: Jossey-Bass, 1993). 
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by OERI. Their work on the acquisition and learning in mathematics 

is highly regarded and was well-received by the outside reviewers 

of their grant application in 1985 and 1990 as well as of their 

continuation grant application in 1987 and 1992. Yet several of 

these same reviewers questioned the quality and utility of the 

Center 1 s work on aspects of social studies such as history. 

Indeed, as one reads through their publications and their 

continuing grant application in 1987 , one is struck by how much 

more simplistic and descriptive some of the social studies research 

is than their mathematical work. 

While some of the research done by the centers is 

sophisticated statistically, others raise questions about its 

methodological rigor. The Center for Research on Evaluation , 

Standards, and Student Testing at UCLA is doing very important and 

thoughtful work on duplex designs which permit the use of more 

complex testing samples while still yielding valid student-level 

scoring. As large-scale educational assessment grows in this 

country, the development and use of duplex sampling will permit a 

85 

broader and more useful employment of test score results. 

On the other hand, some of the work being done at the centers 
is limited by the size of their sample and tie types of statistical 
techniques employed. For example, the studies from the Center for 
the Study of Writing and Literacy at Berkeley often are based upon 



R. Darrell Bock, "Duplex Design: Giving Students a Stake in 
Educational Excellence. H Center for the Study of Evaluation, 
Standards, and Student Testing, Final Report (July 1989). 
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analyzing a single individual or just investigating a few 

8 6 

students. Moreover , the analytic techniques used in many of 

their studies tend to be based more on reporting their activities 

or using simple descriptive statistics rather than employing more 

sophisticated statistical procedures (in part, of course, due to 

87 

limits imposed by their small sample sizes) . While there is 

sometimes merit in analyzing an individual or a small group of 

individuals closely, unfortunately such studies by themselves 

cannot answer the questions about how representative their findings 

88 

are for other populations or settings. 



Melanie Sperling and Sarah WarshaueryFreedman, "A Good Girl 
Writes Like a Good Girl: Written Responds and Clues to the 
Teaching/Learning Process," Center for the Study of Writing and 
Literacy, No. 3 (May 1987) ; Anne Haas Dyson, H The Case of the 
Singing Scientist: A Performance Perspective on the 'Stages 1 of 
School Literacy, 11 Center for the Study of Writing and Literacy, No. 
53 (September 1991) ; Melanie Sperling, "Dialogues of Deliberation: 
Conversation in the Teacher-Student Writing Conference, " Center for 
the Study of Writing and Literacy, No. 48 (May 1991). 

87 

Lorraine Higgins, Linda Flower, and Joseph Petraglia, 
"Planning Together: The Role of Critical Reflection in Student 
Collaboration," Center for the Study of Writing and Literacy, No. 
52 (September 1991) ; Anne Haas Dyson, "Unintended Helping in the 
Primary Grades: Writing in the Children's World," Center for the 
Study of Writing and Literacy, No. 2 (May 1987). 

88 

Despite the statistical limitations inherent in much of 
their studies, the Berkeley Center's own summary of their five-year 
work and achievements does not provide the appropriate caution 
about their findings. Rather, their conclusions are presented in 
a much more definitive way despite the fact that many are based on 
such limited samples. Sarah Warshauer Freedman, "Final Report of 
the Center for the Study of Writing" (December 1990) . 

The use of case studies can be an important and invaluable 
component of an overall research strategy, but efforts should be 
made to develop findings which can be applied to other settings as 
well. For a discussion of the use of case studies, see Joe R. 
Feagin, Anthony M. Drum, and Gideon Sjoberg, eds., A Case for the 
Case Study (Chapel Hill, NC: University of North Carolina Press, 
1991); Charles C. Ragin and Howard S. Becker, What is a Case? 
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Another problem with some of the studies done at the centers 
is that they are based on convenience samples rather than more 
representative ones. Thus, the UCLA Center, which has produced 
some of the more rigorous studies, analyzed the effects of tests on 
what teachers do in classrooms by analyzing a questionnaire filled 
out by 85 elementary and secondary school teachers who attended a 
teacher leadership institute. While the author admits that this 
investigation may not be representative because of the small sample 
size, she does not consider any potential bias that may have been 
introduced as a result of only looking at individuals who attended 

a teacher leadership conference and who volunteered to fill out the 

89 

quest ionna ire . 

Similarly, often it is not clear whether new ideas or products 
are being tested on a representative group of teachers or only on 
a self -selected subset of those who have volunteered to participate 
in this particular activity. For instance, the Pittsburgh Center 
is disseminating and evaluating its Math 3 Program to 37 additional 
teachers. Unfortunately, rather than being able to assign the 
Math 3 Program randomly to teachers, they had to rely upon those who 
volunteered to participate in this rather intensive two-year 

Exploring the Foundations of Social Inquiry (Cambridge: Cambridge 
University Press, 1992) . 

QQ 

Joan Herman, "The Effects of Testing on Teaching and 
Literacy, " Center for the Study of Evaluation, Standards, and 
Student Testing, Final Report (November 1990) . The statistics 
employed in this study are not particularly sophisticated either — 
mainly just descriptive statistics and simple correlations. 
Moreover, although the final report mentions 17 tables, they have 
not been included in this report — thus making it more difficult for 
OERI or anyone else to analyze this work. 
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effort. Consequently, the findings from this evaluation will be 
limited by the fact that the results are based upon a special 
subset of teachers who choose to join this project. 

One might also question the particular research design of some 
of the studies. A study of teacher attitudes about different 
teacher evaluation systems, for example, analyzes teachers in 
Florida and Wisconsin. The strategy for comparing the teachers, 
however, might have been improved. Florida teachers were selected 
from the Miami and Tampa areas. The Wisconsin teachers were from 
Madison and Green Bay — seemingly rather different communities than 
those in Florida (perhaps Milwaukee should have been chosen 
instead) . Moreover, the Florida sample included both beginning and 
experienced teachers while the Wisconsin sample only had 
experienced teachers (again an unfortunate development since in the 
Tampa subsample they found significant and complex effects of 
teacher experience on the ratings) . Given the complexity of the 
issues being addressed in this study, their small sample of only 48 
teachers severely restricted the type of controls and statistical 
analyses that could be performed. Thus, this interesting and 
important study by the Rutgers Center could have been considerably 
strengthened through more attention to the initial research 
design. 

Even some of the more analytically rigorous projects have 



Penelope L. Peterson and Michelle A. Comeaux, "Evaluating 
the Systems: Teachers 1 Perspectives on Teacher Evaluation," 
Educational Evaluation and Policy Analysis , 12, 1 (Spring 1990), 3- 
24. 

a 71 
ERIC 



64 

important methodological issues that warrant further attention. 
The "Success for All" program at the Johns Hopkins Center , which 
was discussed earlier, is certainly one of the more carefully 
designed and statistically rigorous efforts among the centers. The 
researchers at the Johns Hopkins Center examined the ability of 
their "Success for All" program to help young students in some of 
the more disadvantaged urban neighborhoods. One of the nice 
features of their work is the decision to select similar schools in 
the same type of neighborhoods as statistical controls. Moreover , 
in voider to deal with any possible problems introduced in the 
schools due to pupils experiencing different rates of placement in 
special education programs , they matched students from the two 
institutions on the basis of their spring standardized test score 
in kindergarten. 

Yet two potentially important statistical problems have yet to 
be fully resolved. First , one of the criteria of selecting a 
school for the study includes taking only those institutions where 
the administration and the teachers agree to participate by a 
secret ballot. Since the control schools do not require such a 
commitment , one suspects that the schools in the study may have a 
somewhat more dedicated administration and teaching staff than the 
control schools. Second , does the matching of the students from 
one school with those in another one on the basis of a standardized 
test score in kindergarten really provide an adequate control for 
any possible differences? To be sure, since students in the 
program schools as well as the control institutions come from the 
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more disadvantaged neighborhoods , some of the potential range of 

differences are minimized. Yet, as several of the outside 

reviewers as well as the staff of the Office of Research (OR) have 

suggested, the criteria for matching may not be sufficient by 

91 

itself to ensure that no errors were introduced into the study. 

Sometimes the studies do not provide sufficient details for 
the reader to assess and interpret the findings. In an analysis of 
history textbooks, for example, one does not know how many students 
were tested or what were their characteristics. Simple 
correlations are provided in the essay, but since we are not 
provided with information about the size of the sample it is 
impossible to know what are the strengths of those associations. 
Moreover, only in discussing this investigation with the authors 
does one discover that the study occurred in a Catholic school — a 
potentially important contextual factor which might have affected 
the results. 92 

Overall, the quality of research and development at the OERI- 
funded centers is uneven. As we have seen, some of the work 
exemplifies excellent social science analysis while some of it 
leaves considerable room for conceptual and methodological 



For an excellent discussion of the problems involved in 
doing quasi-experimental research for policy purposes, see 

Christopher H. Achen, The Statistical Analysis of Quasj- 

Experimental Experiments (Berkeley, CA: University of California 
Press, 1986) . 

92 M. Ann Britt, Mara Georgi, and Charles A. Perfetti, 
"Learning from History Texts: Conceptual and Text Factors," 
Learning Research and Development Center, University of Pittsburgh 
(1992) . 
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improvement. While no effort was made in this assessment to arrive 
at a precise distribution of the quality of the products produced 
by the seven centers in this study, it does appear that on the 
whole they are of a higher quality than comparable efforts at the 
five OERI-funded regional educational laboratories also 
investigated (though the research or development work from a 

particular lab such as Far West is as good or better than those 

93 

from most of ti*e centers) . 



D. Center Review Process 



As was already mentioned earlier in the report, at various 
periods in the history of NIE, the existing centers were simply 
renewed rather than having to face competition for continued 
funding. In 1985 all of the centers were recompeted and a number 
of them lost their funding to outside competitors. Analysts who 
investigated the process of the center competitions in 1985 as well 
as in 1990 concluded that it was based upon a fair and efficient 
peer review system. 94 Nothing I have seen suggests 



For an analysis and discussion of the quality of research 
or development at the regional educational laboratories, see 
chapter 2. 

94 Chester Finn, "Strengths (and Weaknesses) of Peer Review, " 
Educational Researcher (August/September 1986) , 14-15; Laurie 
Garduque and David C. Berliner, M Beyond the Competition, 11 
Educational Re searcher , 15, 6 (1986), 19-20? Hunter N. Moorman and 
Thomas G. Carroll, M Peer Review and NIE/OERI Competition for 
Regional Educational Laboratories and National R&D Centers , 11 
Educational Researcher (August/ September 1986) , 16-18; Gerald 
Sroufe, "Educational Enterprise Zones: The New National Research 
Centers," Educational Researcher , 20, (1991), 24-29. 
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otherwise. Indeed, it appears that in recent years the Office 
of Research (OR) has been so anxious to avoid any appearance of 
favoritism or bias that the center receiving the highest peer 
review ranking has always received the award — even when that meant 
that the recommendations of the OR staff people most directly 
involved in the process had to be ignored in the case of two 
closely ranked center proposals. 96 

We have already seen how over time OERI has been increasing 
its emphasis on dissemination rather than research — a pattern 
reflected in the 1990 guidelines for center competition. For 
example, on the technical review form for outside evaluators of the 

R&D centers, only 20 points out of a possible 100 points were given 

/ 

"to "technical soundness". On the other hand, 15 points was for 



There has been a recent complaint, by one of the 
unsuccessful former centers, that the winning proposal gained an 
unfair advantage by submitting a longer than allowed proposal. As 
that discussion is still underway, I cannot comment on it except to 
say that there does not appear to be any suggestion that the staff 
of the Office of Research (OR) deliberately favored one institution 
over another in this process. 

96 Staff input in the decision making process was excluded 
because they were not allowed to be part of the formal review 
panel. While this system may be good in that it almost leaves the 
entire decision in the hands of outside reviewers, it is bad 
because it may exclude entirely some of the more knowledgeable and 
experienced people in the field. 

97 Among the criteria for technical soundness was "(1) The 
applicant demonstrates a thorough knowledge of current research and 
development concepts, theories, and outcomes and relates these to 
the proposed activities and mission of the center; (2) the adequacy 
of the research design and methodologies to address the research 
questions posed; and (3) evidence that, where appropriate, the 
perspectives of a variety of disciplines are used." Office of 
Research, OERI, "Application for Grants Under the Educational 
Research and Development Center Program" (Washington, DC: U.S. 
Department of Education, 1990), p. 139. 
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"mission and strategy" , 15 points for "institutional capacity," and 

25 points for "plan of operation" (the latter criteria did not 

include anything about research, but did stress national leadership 

and dissemination) . Indeed, in the 1990 grant application package, 

the 10 additional discretionary points available were all allocated 

to the Plan of Operation "to emphasize the importance of each 

98 

Center's dissemination activities„" In other words, the 

reviewer 1 s technical form did not provide that much incentive for 
or differentiation among proposals in terms of research design. 

While applicants were encouraged to provide detailed research 
designs and reviewers often rewarded those that did, sometimes it 
appears that having a detailed design may not have been especially 
advantageous. In the competition for one particular center, it 
appears that the center which did not spell out its research in 
great detail for the entire five-year period may have in part 
benefited compared to the one that did. Moreover, in many of the 
center competitions some of the reviewers have provided an 
excellent, detailed critique of the research, but then it appears 
that the reviewer 1 s scores do not fully reflect those comments. In 
other words, sometimes there is less variation in the scores 
assigned to the "technical soundness" section than one would have 
expected on the basis of the differences in the written comments. 

One of the major differences between successful center project 
research proposals and successful field-initiated research 
proposals may be that the latter usually would have to provide a 

98 Office of Research, "Application for Grants," p. 138. 
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much more detailed research design than the former. Center 
research project proposals are often brief summary descriptions 
rather than detailed research plans. Some reviewers of center 
proposals have complained that it is difficult to evaluate their 
scientific validity and feasibility because of inadequate 
information about the specific research designs. Center project 
proposals often emphasize the strategy or goals of the research 
effort rather than the details about how they will be carried out. 
Field-initiated research proposals , on the other hand, usually are 
expected to provide short, critical reviews of the literature as 
well as detailed research plans. Moreover, because center research 
projects are put forth as a package rather than as individual 
components, in practice this means that some individual research 

projects are funded which would not have been acceptable by 

99 

themselves • 

Another frequently mentioned criticism of the center review 
process is that the outside panels consist* of both researchers and 
practitioners. No one is saying that the practitioners should be 
excluded entirely from the assessment of the center proposals. But 
some contend that researchers should first evaluate the technical 
quality of the proposals (since many of the practitioners may not 
have had sufficient social science training to properly judge the 
research design and methodology) and only then should the 

99 This comparison of successful center proposals to field- 
initiated research proposals is based upon my personal experiences 
in evaluating grant applications from other agencies such as NEH, 
NIH, and NSF. It is not based upon an analysis of the field- 
initiated proposals submitted to OERI. 
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practitioners and policy makers play a role in the final selection. 
While this proposed two-tier process for selecting centers is 
certainly defensible and perhaps even the way to proceed, the 
mixing of researchers and practitioners on the review panels does 
not appear to have played a major role in the outcomes of the seven 
centers studied. There were relatively few differences between the 
researchers and the practitioners on how they scored the successful 
applications. 100 Perhaps part of the explanation for the 
relative lack of differences between research and practitioner 
reviewers is that so. little weight is given to "technical 
soundness 1 * that it minimizes the variation that could be introduced 
for the overall score on the basis of this item. 

The extent and the quality of review comments for both the 
initial center proposals and their continuation applications varied 
greatly. Some of the reviewers provided very thoughtful and 
constructive assessments while others do not appear to have devoted 
much time or energy to this activity. Since OERI does not pay 
reviewers even a modest honorarium for their efforts, it is not 
surprising that many of the responses are so limited. Moreover, 
the OERI center monitors report having great difficulty in 
persuading scholars and practitioners to participate in the review 
process. Perhaps a modest honorarium of $200 to $500 for 

100 Naturally, a more thorough and proper examination of this 
question would entail looking at all applicants — not just the 
successful one. It is possible that the researchers and the 
practitioners differed on the unsuccessful applicants so that the 
only ones which won were those that were acceptable to both 
parties. 
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participating in the review process would have a positive affect on 
both getting reviewers as well as encouraging them to provide more 
extensive and detailed critiques. 

Although some reviewers of proposals and continuing 
applications do an excellent job overall, many (if not most) are 
rather deficient in discussing in any detail questions about the 
methodology, budget, or the staffing of the research proposals, in 
part this is due to the fact that many proposals do not provide 
detailed information about their research designs for the 
individual projects. As a result, center monitors as well as the 
centers themselves are denied the opportunity to find out 
potentially important, constructive information about these 
research projects. Perhaps the review forms might be slightly 
expanded or made more specific in these areas in order to elicit 
more useful and more detailed comments. Naturally, without 
bolstering the motivation of many of the reviewers through 
something like providing a modest honorarium, it is unlikely that 
any increase in the specificity or breadth of the guestions will 
have much impact. 

Finally, each of the centers is. required to have its own 
evaluation process. Many of the centers distinguish between an 
overall advisory board and a separate process for evaluating center 
products (such as technical papers, publications, etc.). The 
Berkeley Center, for example, effectively uses its national 
advisory board to discuss broader research and dissemination 
issues, but their national board does not attempt to assess the 
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statistical and technical aspects of each research project. The 
Johns Hopkins Center has a general review process for its 
publications , but it does not subject them to an indepth assessment 
of their scientific validity. Instead, the Johns Hopkins Center 
has scholars present their forthcoming work at one of the weekly 
seminars for critical feedback about their methodology and 
interpretations. Un fortunately , not all of the authors are able to 
have their work scrutinized at one of these sessions — especially 
those colleagues who are working as subcontractors at another 
university. Thus, while a few of the centers seem to have a very 
active and useful outside advisory and evaluation groups to help 
them with their research priorities and quality control, many 
others do not appear to use such mechanisms very effectively. 

E. Center Monitoring 



There is considerable variation in how the Office of Research 
(OR) staff monitor and interact with the centers. Each center has 
an OR center monitor assigned to it as well as usually someone who 
can function as a back-up. Each of the center monitors reports to 
one of four division leaders in OR (the Learning and Instruction 
Division [Anne Sweet] ; the Schools and School Professionals 
Division [Hunter Moorman]; the Higher Education and Adult Learning 
Division [Clifford Adelman] ; and the Education and Society Division 
[Ivor Pritchard]). In addition, there is a Director of Center 
Management and Operations [Ned Chalker] who has a small staff for 
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assistance. Both the division heads and the Director of Center 

Management and Operations report to the Director of the Office of 

101 ♦ 
Research [Joseph Conaty] . At the present time, this 

arrangement is informal and the lines of authority and the division 

of responsibilities between the division heads and the Director of 

Center Management and Operations are not always clear — though in 

practice they seem to be able to work together quite harmoniously. 

One of the problems in the current system of monitoring is 

that there may be some variation in how individual monitors handle 

or respond to a particular situation. While the division heads and 

the Director of Center Management and Operations try to co-ordinate 

the activities of the center monitors, in practice there is still 

room for some differences. The problems are compounded by the fact 

that some of the center monitors have only short-term appointments 

in OERI so that there is a need to recruit and train new monitors 

on a regular basis. Under NIE a special, brief handbook for 

103 

monitors was developed to assist them. Similarly, Sally 

Kilgore, a former director of the Office of Research (OR) and 
others on the staff developed a handbook in 1988 which several of 
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101 When this study began, Milton Goldberg was the Directcr of 
the Office of Research. He was replaced in July 1992 when he left 
OERI for another assignment. 

102 Sometimes there has been an excessive turnover in center 
monitors that has been detrimental both to the centers and OERI. 
For example, the former National Center on Education and Employment 
had five different center monitors in a 2 1/2 year period. 

103 Susan Gruskin, "Monitoring Manual," National Institute of 
Education (August 1975) . 
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the current monitors found helpful when it was used five years 
104 

ago. Unfortunately, that handbook was allowed to become 

dated and obsolete over time so that today there is nothing 

105 

comparable for new or continuing center monitors today. 

Part of the reason for the diversity of practices among center 
monitors is because the division heads and the monitors do not 
agree amongst themselves on the importance or the nature of this 
responsibility. Some division heads and center monitors, for 
example, see their relationship with the center*, as a very active 
one that requires considerable time and effort. Other division 
heads and center monitors see the centers as just another set of 
grantees that do not need or deserve intense scrutiny and 
involvement. Moreover, there appears to be some differences among 
the recent Assistant Secretaries of Education on just how involved 
centers monitors should be or the types of activities that they 
should perform. For example, while some staff have seen themselves 
as liaisons to the centers, Diane Ravitch, the recent former 
Assistant Secretary, prefers to see them as center monitors. 

Center monitors also vary considerably in terms of their 
training and orientation. While a few center monitors have only 
had some college education, others have received a Ph.D. or E.D. 
degree. Some have had extensive personal experience in conducting 



104 El izabeth Ashburn et al . , "A Handbook for OR Center 
Liaisons," Office of Research (August 1988). 

105 Even at the time when Sally Kilgore tried have center 
monitors use the handbook, one of the division heads simply refused 
to comply and told the monitors reporting to him to ignore the 
manual • 
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or supervising research while others have had relatively little 
previous exposure to research. Similarly, while the training and 
experiences of some monitors fits very nicely with the subject 
matter of their center, others are working in areas that are far 
removed from their previous expertise or interests. A few center 
monitors are so deeply involved and knowledgeable about the content 
of their center's activities that they have even co-authored work 
with the personnel of that institution. 106 But some centers 
complain the OR center monitors are not familiar about their 
center's work or knowledgeable enough about academic research to be 
of much assistance. 

There is also considerable variation in how much monitors 
consider and discuss the research design and methodology employed 
in the center's individual research projects c Some monitors devote 
relatively little time or effort to look at these issues while 
others are very deeply involved in such discourse. For example, 
the monitor for Johns Hopkins Center raised questions about the 



Judith Segal, for example, has co-edited several books 
wxth scholars at the center that she monitors. Judith W. Segal, 
Susan F. Chipman, and Robert Glaser, eds., Thinking and Learning 
Skills: Relating Instruction to Research (Hillsdale, NT: Lawrence 
Erlbaum, 1985); Susan F. Chipman, Judith Segal, and Robert Glaser, 
eds . , Thinking and Learning skills: Research and Open Questions 
(Hillsdale, NJ: Lawrence Erlbaum, 1985) ; James F. Voss, David N. 
Perkins, and Judith W. Segal, eds., Informal Reasoning and 
Education (Hillsdale, NJ: Lawrence Erlbaum, 1991) . Of course, 
while the close intellectual involvement of a monitor with their 
center is to be welcomed and fostered, someone might object that it 
diminishes from the possibility of that monitor maintaining an 
independent and objective perspective. While such a concern is 
legitimate, such dangers in practice are probably far outweighed by 
concerns about not having a center monitor who is intimately 
familiar with the subject and methodological issues involved. 

eric 
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issue of matched sampling for the "Success for All" program and 

tried unsuccessfully to persuade the researchers there to be more 

concerned about this issue. Similarly, the monitor for the 

Berkeley Center questioned various aspects of the research design 

and strategy for that institution. 

One major problem for center monitors is that they often do 

not have sufficient travel funds to visit their centers or to bring 

with them, if necessary, the type of experts necessary to evaluate 

107 

the research being produced by their center • Under NIE in 

the 1970s, for example, center monitors sometimes were able to 
assemble teams of experts to go with them on their center visits. 
As OERI becomes more concerned about improving the quality of the 
research being done by the centers as well as assisting them with 
their other activities, it may be useful to provide more funding 
for travel and consultants. 

Finally, some of the monitors pay close attention to the 
budget from a programmatic perspective, but others do not. Those 
who are interested in looking closely at the budget information, 
especially from a comparative perspective, are handicapped by the 
fact that the results from the computerized Project Management 
Information System (PMIS) do not provide the necessary information. 



Indeed, since most center monitors are only allowed one 
annual trip either to their center or to a professional conference, 
some monitors choose to attend the annual conference and meet with 
the center director there. Center monitors should not be placed 
into this unfortunate trade-off. Travel funds should be available 
for their center monitoring activities at the site as well as for 
an opportunity to attend an appropriate professional meeting to 
further their own education and training. 

EMC b : 



77 

Moreover, sometimes the appropriate research activities information 

from the Project Input Forms which would be necessary for a more 

indepth budget analysis were sent to OERI, but not passed on to the 

108 

appropriate monitors. In general, one gets the impression 

that the analysis of the individual project research budgets and 
staffs have not received as much attention from either center 
monitors or outside reviewers. 

In 1987 the Office of Research (OR) undertook a third-year 
review of the centers. The control and direction of those reviews 
was placed in the hands of the individual monitors so that the 
quality of those efforts varied considerably from center to center. 
Overall, the reviews for the centers investigated for this study 
appear to have dealt only marginally with the quality of the 
research and development produced by those institutions. 

The third-year review for 17 of the centers in 1992 overall 

was much more rigorous and systematic. The outside reviewers and 

center monitors were specifically asked to comment upon the quality 

of the work produced by the centers during their first two years 

and the results of this review were scrutinized both by the Office 

109 

of Research and the Office of the Assistant Secretary. 
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Keith Stubbs and Ella Jones are now heading up an OERI 
task force to review and revise the use of the PHIS system. 

109 

There was Considerable debate within OR and OERI on how 
much attention should be paid to the quality of the research and 
development produced by the centers. Initially, the third-year 
review plans did not call for a careful assessment of the quality 
of the center w^rk, but the final instructions to the outside and 
inside reviewers specifically asked them to address the quality 
issue. 
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One source of uncertainty among some of the staff in OR is 
over he* much legal authority center monitors have to make changes 
in the grants. Some have said that since the funds are grants 
rather than contracts , OR cannot insist on ma j or changes or 
deletions of entire projects — though the center monitors have often 
persuaded the centers that changes or a redirection of funding are 
warranted and prudent. Others believe that monitors have the legal 
authority to terminate any specific project that is deemed 
inadequate — though usually this would be in the context of 
redirecting those monies to other projects within the same center. 
In any case, it does not appear that until last year OR has 
terminated the funding for a specific project within a center and 
then decreased the overall funding level for that center. 

The Department of Education grant regulations state that 
H [t]he Secretary may make a continuation award for a budget period 
after the first budget period of an approved multi-year project if: 
• •• (4) Continuation of the project is in the best interest of the 
Federal Government." 110 In other words , if OR decides , for 
example, that ultimately one of the projects in a center is 
inadequate from a research perspective or unnecessary from a 
government perspective, it can terminate that project and reassign 
the money to other activities within that center or transfer the 



U.S. Department of Education, fiducacion Department General 
Administrative Reg u lations (EDGAR) t 34 CFR Parts 74, 75, 76, 77, 
79. 80, 81, 82, 85. and 86, Revised July 8. 1992 (Washington, DC: 
U.S. Government Printing Office, 1992), paragraph 75.253. 
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funds to other centers. 111 

In the 1970s, NIE occasionally did remove funds from a 
particular center project or projects and then use them for some 
other purposes. But in more recent years OR has not followed that 
same practice — perhaps in part because some of the monitors as well 
as some of the leadership in OERI was not aware that this could be 
done. In the third-year review of the centers, however, a specific 

center project was defunded and the monies were used for other 

112 

center expenditures. The defunding of specific center 

projects which are deemed by center monitors and reviewers as 
unworthy and unsalvageable should be pursued as a last resort, at 
least on a limited basis, because it sends a clear message to 
everyone that OR is only willing to fund those projects which are 
of demonstrable quality and "in the best interest of the Federal 
Government . " 

There is another major problem with the relationship between 
the centers and the Office of Research (OR) . The research produced 
by the centers is transmitted to the monitors in OR, but there is 
no real effort made to incorporate and use their findings in the 
Office of Educational Research and Improvement (OERI) . Basically, 
up until very recently, few individuals within OR have been able 



Since the Congress has specified the total minimum amount 
of money to be spent on the centers as a line-item, the surplus 
funds cannot be transferred to some other non-center activity at 
OERI (unless the total amount allocated to the centers by OERI had 
exceeded the minimum set by Congress) . 

112 Similarly, over $200,000 of another center 1 s funding was 
withheld until it provided a much more detailed and developed 
research design for one of its projects. 
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to do any serious original research or to synthesize existing 
studies. Moreover, the lack of subject matter specialists in OR 
like someone in early childhood education have made it difficult 
for the Office of Research to interact intellectually with many of 
the centers. 

The lack of research personnel and activity within the Office 
of Research seriously weakens that group's ability to monitor and 
interact substantively with the centers, in recent years this 
situation has become worse as personnel have been shifted from OR 
to other tasks within OERI. The number of full-time equivalent 
(FTE) employees in the Office of Research decreased from 68 in FY87 
to 58 in FY91-- a fifteen percent decrease. As a result, the 
ability of OR staff to do original analyses or to synthesize the 
research findings from the centers has decreased at the same time 
that the total number of centers funded by OR has increased, 

Diane Ravitch, the recent former Assistant Secretary, agreed 
that the Office of Research has been under-staffed and she 
authorized in late 1992 the addition of four more researchers to 
the staff. It was envisioned that the new staff members would be 
part of a separate unit within OR which would specialize in 
synthesizing existing research as well as undertaking their own 
investigations. As Ravitch f s tenure in office was terminated by 
the change in administrations, her proposed expansion of OR has 
been placed on hold. 
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The R&D centers were initially created as large, national 
research institutions to develop an ongoing and cumulative program 
of education research on some vital problem area. While that need 
continues to exist, the size and nature of most centers has changed 
dramatically so that it will be necessary to make some major 
changes in our current center system in order to meet those initial 
expectations. 

Most of the initial R&D centers in the early 1970s were $2-$4 
million operations (in constant 1982-84 dollars) and many of the 
previous analyses of these institutions recommended that they 
should be even larger. Today, the average size of the R&D centers 
is only about three-quarters of a million dollars (in constant 
1982-84 dollars) and they cannot be considered truly "national" 
research centers in the original meaning of that word. Instead, 
most of them are small research institutions (or coalitions of 
smail research institutions) which do not have the national 
presence or impact that was initially envisioned. If the R&D 
centers are to fulfill their functions of supporting sustained and 
cumulative research on a national scale, they will have to be much 
larger — perhaps in the range of $6 to $8 million (current dollars) . 

The total amount of money in constant dollars allocated to R&D 
centers has declined substantially from a high of $27.1 million in 
FY74 to $15.6 million in FY91 (in constant 1982-84 dollars) — a 42.4 
percent decrease. The recent National Academy of Science Report is 
correct in recommending that there should be a sizable increase ii* 
total center funding over the next few years (they recommended 
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approximately doubling the total R&D center budget) . 

But the R&D centers are only one component of the overall 
research portfolio of the Office of Educational Research and 
Improvement (OERI) . Equally important are mission-oriented basic 
research or field-initiated studies. If the U.S. Congress does not 
substantially increase the funding for these other activities, it 
may be advisable to redirect some of the other funds within OERI 
for those purposes. Therefore, if there is no increased funding 
for mission-oriented basic research or field-initiated research, 
regrettably it may be prudent to redirect perhaps 10-15 percent of 
the current center funding for these activities (this would require 

. 113 

congressional approval) . 

More of the current budget of the centers needs to be directed 
toward research. The fact that only approximately half of the 
funds allocated to the five centers studied went directly to 
research projects is troublesome. The federal government needs to 
reconsider the indirect cost rates of colleges and universities to 
make sure that they include only the real costs of providing 
services for conducting research in education at the centers. More 
attention needs to be paid to the administrative costs of the 
centers charged to the OERI grant. Centers whose universities 
charge a relatively high indirect cost presumably will not need as 
much additional help from OERI for administrative expenses. In 
order to interact more thoughtfully with center budgetary matters, 

113 

The same percentage of money should be redirected to the 
field-initiated or mission-oriented basic research from OERI's 
funding of the regional educational laboratories. 
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it will be necessary to expand and improve the output from the 
current PMIS system. The computerized output provided by the 
present PMIS system has not been of much interest or help to the 
center monitors or other program analysts within OERI. 

The cost of disseminating information from the centers may 
need reconsideration. There is no question that the dissemination 
of information about the findings from center research projects is 
an important and necessary activity. In the present situation of 
mostly small R&D centers, however, sometimes the costs of 
dissemination in a particular center are too high relative to the 
expense of doing the research itself so that one wonders if the 
proper balance has been found. In an extreme case, one of the 
centers spends more money on collaboration and dissemination than 
it does on research. If the centers become larger, then the 
seemingly direct trade-off between dissemination and research will 
not be as problematic. In any case, centers should also work much 
more closely with other institutions, such as the laboratories, in 
order to reach a broader audience in a more efficient and cost- 
effective manner. Indeed, it was rather disappointing to discover 
how little co-ordination and co-operation exists between most 
centers and labs today. 

Another problem with the current distribution of research 
activities within the centers is the relative lack of basic 
research, As the recent National Academy of Science Report pointed 
out, very little basic research is being done in OERI and as this 
analysis has confirmed, very little basic research is being done at 
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the centers. This is unfortunate because so much of OERI's very 
limited research funds are expended through the centers and labs. 
While more basic research can and should be expanded through field- 
initiated research and other more focused research grants, the 
centers should expand their relative commitment of time and money 
to basic research — especially since the work of the centers is 
supposed to be more sustained and cumulative than ordinary research 

grants anyway. Therefore, mission-oriented basic research should 

114 

be expanded within the centers. 

If one of the major rationales for the centers is the need for 
co-ordinated and cumulative research strategies, then they should 
strive to accomplish this goal. At the present time, many of the 
projects within the centers could not be justified on these basis. 
Given the current small overall size of the centers, the decision 
to put together a loose coalition of colleges and universities may 
not always be wise as it often leads in practice to further 
fragmentation. If the centers were to increase in size, then the 
idea of bringing together different institutions might become more 
attractive as each one would receive more funding and as there 



Some have suggested that as there are fewer and larger 
centers, the centers themselves should hold the mission-oriented 
basic research competitions for individual scholars. Perhaps. But 
it may be more effective and efficient to have the Office of 
Research hold those grant competitions than the centers. This 
important issue should be analyzed further before any decision is 
made. Part of the answer to this question will depend on how the 
congressionally proposed institutes or directorates are organized 
and run. 
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might be more monies for travel and co-ordination. 

If spreading the center among several different universities 
often may be problematic, so too is the almost universal practice 
of having many small research projects. Despite the small size of 
the current centers, almost all of them have decided to fund 10-35 
research projects. As a result, almost by definition most of these 
research projects are small and limited efforts which make it 
difficult, if not almost impossible, to provide the type of large- 
scale and long-term sustained research analyses initially 
envisioned by the creators of these institutions. 

With some notable exceptions, many of the center activities 
are more a series of interesting and often well-done individual 
research projects rather than part of a closely co-ordinated, 
ongoing research strategy. Sometimes there is considerable overlap 
among the centers in their research projects and orientations — too 
often without any close co-ordination among them. Even those 
projects which are part of a broader research strategy sometimes 
appear isolated and unrelated to outsiders who do not have direct 
access to the researchers or their unpublished proposals to OERI. 
In short, if the centers are to be continued because of the need 
for sustained and cumulative research, then they must organize 
themselves in such a manner as to be able to deliver it — otherwise 
they may be simply perceived as expensive and privileged 
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While several of the centers which are composed of 
coalitions of colleges and universities were not particularly 
effective or useful, some were more functional. For example, the 
Rutgers Center made effective use of faculty at different colleges 
and universities. 
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institutions for catering to the particular individual research 
interests of their faculty members. 

While everyone agrees that center grants should be 
competitive , there is considerable disagreement over how frequently 
the existing centers should be recompeted. At the present time, 
all centers face a new round of competition every five years. The 
recent National Academy of Science Report, however, suggests that 
center grants should be made for 10 to 15 years — conditional, of 
course, upon satisfactory annual progress. 

As desirable as it may be to have more long-term centers, it 
is probably wise to continue to recompete them every five years. 
The frequent competition provides additional incentives for 
existing centers to maintain their research quality and provides 
more opportunities to redirect the research focus of portions of 
the center system if chat should ever become necessary. 

Perhaps one way of combining the desire for more stability 
with the emphasis on the need for quality is to allow center 
recompetitions to take into consideration the prior work of the 
incumbent center. Centers which have done first-rate research 
would gain some advantage in the recompetition because of their 
existing tracx record. This might not necessarily be undesirable 
from a government perspective because centers are supposed to 
develop ongoing cumulative education research. On the other hand, 
incumbent centers whose research has been conceptually and 
methodologically flawed would be disadvantaged — again not an 
undesirable outcome. Taking into consideration an incumbents past 
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performance could play an important role in both enhancing the 
overall quality of center research as well as providing more 
stability for those institutions which produce high quality and 
useful work. 116 

Another way to enhance the long-term impact of the centers is 

to require that OERI funded data produced at those institutions be 

made available to other scholars and analysts within a relatively 

short time-period (perhaps one year after the data has been 

gathered and first analyzed) . The National Science Foundation 

already requires all of its grantees to provide public use computer 

tapes/disks and OERI should adopt their policy. Naturally, the 

confidentiality of the research subjects must be protected and 

specific funds should be set aside in the center budgets for the 

117 

cost of preparing a public use computer tape/disk. 

The overall quality of the research produced by the centers is 
mixed. On the one hand, some of the projects within centers have 
produced excellent social science research. Their investigations 
are well-designed and their studies employ sophisticated research 
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Taking into account an incumbent center 1 s previous work is 
certainly permissible under existing U.S. Department of Education 
regulations. In describing how the Secretary selects applications 
for new grants, the regulations state: "(3) any other information 
relevant to a criterion, priority, or other requirement that 
applies to the selection of applications for new grants, including 
information concerning the applicant's use of funds under a 
previous award under the same Federal program." U.S. Department 
of Education, EDGAR , paragraph 75.217. 

117 OR has created a task force to look into the matter of 
centers providing public use data tapes. OR has already moved to 
obtain such data from the Johns Hopkins Center which is now 
entering its final year of OERI funding. 
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techniques ranging from insightful ethnographic analyses to 
rigorous statistical methods. On the other hand, some of the 
research projects are so conceptually and methodologically weak 
that they either should have never been funded or should have been 
promptly improved after being funded. While there is considerable 
variation among the centers in the quality of their work, there is 
also a surprisingly large variation in the quality of work within 
many centers. Even at some of the best R&D centers there are 
research projects which probably would not have been funded if they 
would have had to compete openly by themselves for scarce federal 

research dollars. 

It is evident that first-rate educational research has been 
and can be produced through a system of R&D centers. It is also 
clear that insufficient attention has been paid by the Office of 
Research (OR) to ways of improving the overall quality of research 
from the centers—though individual center monitors often have 
worked very hard to help maintain high research standards at their 

centers. There are a number of steps which might be taken to 

118 

enhance the quality of the research produced by the centers. 

More attention needs to be paid to the research design in the 
original grants. In 1985 i^nd 1990 only 20 percent of an applicants 
overall score depended upon their research design—a much too low 
percentage for centers which are explicitly designed for research 



118 Starting with the third-year review of the 17 centers in 
1992, OR now has initiated a series of efforts to monitor and 
improve the quality of the research and development produced by the 
centers. These recent activities by OR are unique historically and 
the office should be commended for its initiatives in this area. 
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and development purposes. Applicants for center grants must be 
explicitly and forcefully reminded to present more specific and 
detailed research plans for each of their projects (even if this 
means allowing a few more pages for each individual project 
description in the application) . If the panel of reviewers feels 
that any particular project of the winning application is so weak 
conceptually or methodologically (or so unclear because they have 
not been given sufficient information) , they either should delete 
it immediately from the approved package or clearly indicate that 
the project must be improved quickly. Once applicants for center 
grants realize that their individual research projects might be 
entirely eliminated if their proposals are not conceptually and 
methodologically sound and clearly presented, perhaps the overall 
quality of the successful applications will be improved. 

It is essential to obtain good reviewers for both the initial 
grant review panel as well as for those participating in the third- 
year continuation review. Highly qualified researchers should play 
a prominent, though not exclusive, role in these deliberations. 
Whether the review system becomes a two-tiered process (with 
practitioners or policy makers and researchers separated) may be 
less important than whether the panel (s) are able and willing to 
discuss thoroughly the research designs of each project. One of 
the problems, however, of including practitioners, policy makers, 
and researchers on the same panel is that there may not be enough 
positions for covering the broad array of substantive issues being 
raised — especially if the centers become much larger in scope and 
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function. In order to recruit highly qualified reviewers and 

persuade them to devote the amount of attention suggested here, a 

119 

modest honorarium should be provided. Given the sizable 

public investment in each of these centers, it is wasteful and 
inefficient to maintain a review process that cannot ensure a high 
degree of scrutiny of the soundness of the proposed research in 
these centers. 

Center monitors as a group need to devote more attention to 
the research quality of the products from their centers. This 
means that center monitors either should be personally competent to 
assess the quality of the research or able to seek assistance from 
colleagues within the Office of Research (OR) . Just as center 
monitors now may receive technical training in how to handle grants 
and contracts or how to use computers, they should also have the 
opportunity to receive some training in research design and 
statistical analysis. This is particularly important in OR because 
some of the staff have risen through the ranks without ever having 
had the type of methodological training that they may need in the 
current positions. 120 More efforts also have to be made to 

119 OERI has just changed its policies in regard to paying 
reviewers. If funding permits, OERI now will try to pay reviewers 
an honorarium of $100 per day — a practice consistent with the 
Departments current practices. Dick Hayes, M Memo to Program 
Directors, H Office of Educational Research and Improvement (May 20, 
1993) . 

120 Some of this training is now being provided to OERI staff 
by the National Center for Education Statistics (NCES) . The new 
NCES social science training program is an important step forward 
in this area for all of OERI. 

In addition to the offerings by NCES, the Horace Mann Learning 
Center for the U.S. Department of Education also should provide 
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standardize some of the routine center monitoring practices in OR; 

the development and annual updating of an OR handbook for center 

121 

monitors would be a step in the right direction* 

Travel money should be made available for center monitors to 
visit their sites on appropriate occasions. Center monitors should 
not be forced to decide between attending one professional meeting 
each year or visiting their center as the current policy dictates. 
Moreover, funds should be available occasionally to bring in 
outside consultants to visit a particular center for a more indepth 
assessment of its strengths and. weaknesses. 

Center monitors should be encouraged to continue working 
closely with their centers to improve the guality of research. 
Better individual project forms for obtaining more detailed 
information about research design and methodology need to be 
developed and employed — perhaps along the lines of those now being 
created for the TORUS System. Conceptually and methodologically 
weak projects should be improved or the funds allocated for them 
redirected to other center projects. In some cases it may be 



more social science training opportunities for federal employees. 
At the present time the Horace Mann Learning Center concentrates on 
computer training and technical assistance for grant and contract 
management. It also sponsors some other more controversial 
activities such as an image impact seminar to "help participants to 
understand the importance of projecting the right image in business 
through appearance, attitude and body language." U.S. Department 
of Education, Training Announcement, "Image Impact Seminar," Horace 
Mann Learning Center, Announcement No. 92-125 (December 22, 1992). 
Unfortunately, since the image impact seminar is explicitly 
targeted only for clerical/secretarial employees, it may raise some 
questions about occupational stereotyping. 

121 OR is now in the process of developing a new handbook for 
center monitors. 
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necessary and desirable to eliminate a particular center project 
entirely and then reduce the overall funding of that center 
accordingly. Any surplus funds accumulated from research projects 
eliminated from a particular center might then be reallocated to 
other worthy research projects at other centers. 

The Office of Research (OR) needs to conduct its own research 
as well as to synthesize the research findings from the centers. 
At the present time, the staff of the OR are not given sufficient 
time to produce or synthesize research. Part of this inattention 
to research within OR is due to a general de-emphasis on the 
importance of research within OERI until very recently and part is 
due to past decisions to reduce the overall staff size of OR 
(fortunately the current plans call for an expansion of OR analytic 
staff) . At the present time there is a serious shortage of 
qualified analysts within OR who are able to address the various 
substantive topics pursued at the existing centers. Therefore, 
additional high quality researchers in OR who have expertise both 
in substantive and methodological areas should identified and 
hired. If the work produced by the centers is expected to provide 
a sustained and cumulative body of educational research, then the 
research staff within OR will substantially have to be expanded in 
numbers and upgraded in quality. 

While there has been some discussion and consideration at OR 
of the role of research at the centers, very little, if any, 
attention has been paid to the issue of development — despite the 
fact that this word is prominently highlighted in the formal 
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designations of the centers. The relative lack of developmental 
work in OERI in general is a serious problem for the agency and one 
that needs to be investigated and remedied. Whether that 
devfc 1 opment shou Id be done by the cent ^r s , 1 abs , or some 
combination of these and other institutions remains to be explored. 
In any case, the Office of Research (OR) should carefully 
reconsider the place of development in the functions of the centers 
and either drop that word from the title and descriptions of the 
centers or deal with this issue more seriously in practice. At the 
present time, it appears that many, if not most centers, do not 
pursue their developmental responsibilities very diligently. 



Chapter 2 



ANALYSIS OF THE QUALITY OF RESEARCH AND DEVELOPMENT 
AT THE OERI REGIONAL EDUCATIONAL LABORATORIES 
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In the previous chapter we discussed the research and 
development (R&D) centers of the Office of Educational Research and 
Development (OERI) — one of the major areas of research expenditures 
by that agency. Now we will turn to an investigation of the types 
and quality of research and development at the regional educational 
laboratories, the other major OERI-funded institution created just 
two years after the R&D centers were established. From 1966 
through 1991, approximately $811 million (in constant 1982-84 
dollars) have been expended on these labs — about 30 percent more 
than spent on the R&D centers from 1964 through 1991. 1 Similar to 
the situation with the R&D centers, the characteristics and quality 
of the research and development produced at the labs have not been 
systematically investigated. 

This study will analyze the nature and quality of research and 
development produced by the labs since 1985. Compared to the 
earlier study of research and development at the R&D centers, this 
undertaking is more complicated because the mission of the labs has 
not always been as directly or explicitly focused on research and 
development. Therefore considerable care and attention will be 
paid to the varying definitions of what constitutes research and 

1 The information on lab funding is based upon data from a 
variety of different documents gathered at OERI. While there are 
a few minor discrepancies in the various annual estimates of lab 
expenditures, the overall level as well as trend should be 
reasonably accurate. The cost of living information is from the 
U.S. Department of Commerce index of consumer prices. 



development activities at these institutions as well as to 
assessing the quality of their work. Again, there will be some 
need to look at the changing overall trends in lab budgets and to 
examine the relative amounts expended on research and development. 
Since the focus of this inquiry is an assessment of the quality of 
research and development produced at the labs rather than their 
impact on schools or the role of the labs in providing technical 
assistance to states and local communities, this analysis should 
not be seen as a comprehensive investigation of the overall 
functioning of the labs. 

The study begins with an analysis of the creation of the labs 
in 1966 and their subsequent development. Because there were so 
many major changes in the focus and m direction of the labs, the 
historical analysis will be examined in considerable depth. 
Readers only interested in the current functioning of the labs 
rather than their origins or historical development might want to 
skip this s* ^tion and go directly to the analysis of the quality of 
research and development work at five of the current labs. Then 
the relationship between the labs and the staff at OERI will be 
considered in an effort to make some suggestions about how to 
improve the quality of research at those institutions. 

I. Historical Development of the Labs 

In the previous chapter, we described the growing role of the 
federal government in the support of educational research in the 
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late-1950s and early-1960s and the frustration over the fact that 

rouch of that initial research was piecemeal and fragmentary. The 

R&D centers first were established in 1964 as part of a concerted 

effort to develop sustained and cumulative educational research and 

the centers were reauthorized under Title IV of the Elementary and 

Secondary Education Act (ESEA) of 1965. 

President Lyndon Johnson created in 1964 a number of task 

forces to consider what actions the federal government should take 

in different areas. The task force on education was headed by John 

Gardner and it played a key role in the establishment and operation 

of the educational laboratories. Whereas the Gardner Task Force 

virtually ignored the role of the newly created R&D centers, it 

made the creation of the educational laboratories one of the 

centerpieces of its recommendations for bold new ways of improving 

learning. Moreover, in its relatively extensive discussion of the 

laboratories, it set forth a particular vision of those 

institutions which provides some of the most detailed information 

2 

about the original intent of the lab founders. 

The Gardner Task Force, in language almost identical to some 
of the current rhetoric of reform, called for "a massive burst of 

2 For a useful analysis of the Gardner Task Force, see Charles 
Philip Kearney, "The 1964 Presidential Task Force on Education and 
the Elementary and Secondary Education Act of 1965." Unpub. Ph.D. 
diss., University of Chicago, 1967. At the time that Kearney did 
his analysis, he did not have a copy of the official report of the 
Gardner Task Force Report, but he was able to recreate the contents 
of that document from materials supplied by some of the 
participants. Indeed, his understanding of the labs based upon 
earlier documents as well as discussions with the participants 
almost exactly mirrors what is found in the final draft of that 
report . 
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innovation." Since they saw educational development as ever 
changing, they called for "a system designed for continuous 
renewal, a system in which reappraisal and innovation are built in. 

That is why references to research and development, to innovation 

3 

and experiment, appear in every chapter of this report. 11 

Laboratories were seen as a way of overcoming the lack of 
adequate dissemination and adoption of new educational ideas and 
practices: 

The past ten years have brought a wealth of new 
ideas and programs in almost every area of education, 
particularly at the elementary and secondary level. Yet 
the efforts of the past ten years have not brought about 
the far-reaching changes that one might wish, partly 
because neither the efforts to innovate nor the 
arrangements for disseminating innovation have been on a 
scale adequate to the need. To remedy that defect, the 
Task Force recommends Federal aid for the establishment 
of large-flcale National Educational Laboratories which 
would develop and disseminate ideas and programs for 
improving educational practices throughout the country. 
There should be at least a dozen major laboratories and 
perhaps two or three dozen more that are specialized or 



John Gardner, "Report of the President 1 s Task Force on 
Education" (November 14, 1964), p. 33. The report is available at 
the LBJ Presidential Library, Austin, Texas. 
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less ambitious in scope* 



The Gardner Task Force explicitly stated that these labs 
should not be "small-scale efforts, operating out of a corner of a 
department of education, rooted in the interests of a few faculty 
members, and having little connection with the daily practice of 
education in the community . " Instead, they saw them as ">aore 

closely akin to the great national laboratories of the Atomic 

5 

Energy Commission and should share many of their features." 
Although they should pay considerable attention to basic research, 
"the central focus of the laboratories will be on the development 
and dissemination of educational innovations." 6 

Given the emphasis on developing and disseminating educational 
innovations to the local schools, the Gardner Task Force emphasized 
the importance of testing and refining materials. Therefore, they 
recommended that each lab have one or more experimental schools 
under its own jurisdiction. "Taken collectively, these 

experimental schools might constitute a nationwide network to test 
the feasibility of new methods. In other words the school would 
not only serve the laboratory with which it was affiliated but 
other laboratories too. Thus it might at any given time be testing 
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Education," p. 34. 
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a new reading unit that had been locally produced and a mathematics 

7 

unit that had been developed 3,000 miles away." They also 

recommended that links be established "with numerous schools (or 

school systems) for the sake of teacher training and the field 

testing of new programs." 

Unlike the situation today, where only one of the ten labs is 

affiliated with a university, the Gardner Task Force argued that 

9 

all labs should have a university connection. "It would also be 

essential that each laboratory have some kind of affiliation with 

a neighboring university. It could be under the jurisdiction of a 

university, or sponsored by a group of universities, or affiliated 

10 

in other ways." 

Although the existing R&D centers^ and the proposed labs might 
overlap to some degree, they were seen as different, but 
complementary entities. " [T]he Laboratories described here go 
beyond the centers created by the Office in certain respects, 
chiefly the following: (a) considerably greater emphasis on 
development and upon the dissemination of innovation, (b) the use 



7 Gardner, "Report of the Presidents Task Force on 
Education," p. 36. 

8 Gardner, "Report of the Presidents Task Force on 
Education," p. 35. 

9 But some of the other laboratories today do try to maintain 
close ties to colleges and universities. For example, the 
Appalachia Educational Laboratory (AEL) has representatives from 
the colleges and universities on their governing board, works with 
them on their Colleges and Schools Program, and hires expert 
consultants from those institutions. 

10 Gardner, "Report of the Presidents Task Force on 
Education," p. 36, 
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of experimental schools and extensive pilot programs in the regular 

schools and (c) provision for teacher training as an integral part 

11 

of the program." 

The ideas of the Gardner Task Force on the educational 

laboratories played a very important role in the development and 

passage of Title IV of the Elementary and Secondary Education Act 

(ESEA) of 1965. Title IV authorized the establishment of a network 

of large-scale labs, but did not require that the labs have 

experimental schools associated with them or that teacher training 

be an integral part of their activities. Moreover, while 

universities could apply for the funds to create these labs, so too 

could other non-profit groups which were not affiliated with 
12 

universities. 

There was considerable disagreement over the size of the 
initial labs and how rapidly they should be created. The Gardner 
Task Force envisioned well-funded, large-scale institutions. Some 
of the lab proponents also argued that these organizations should 
be phased in gradually so that ample time would be allowed to 
experiment with the best institutional arrangements for them. 

A decision was made to create quickly a large number of labs 
and to place them in various regions of the country — partly in the 
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11 Gardner, "Report of the Presidents Task Force on 
Education," p. 37. 

12 Kearney, "The 1964 Presidential Task Force on Education". 
For an a discussion of the Elementary and Secondary Education Act 
(ESEA) of 1965, see Stephen K. Bailey and Edith K. Mosher, ESEA: 
The Office of Education Administers a Law (Syracuse, NY: Syracuse 
University Press, 1968). 
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hope that the U.S. Congress would quickly increase funding. In 
January of 1966 10 labs were funded and by the end of that year, 20 
labs were under contract. Nineteen of the twenty labs were 
designed to serve the states and one, the Center for Urban 
Education, was intended for helping the major urban areas. The 
manner in which regions were drawn meant that some states were 
split between two or more labs; on the average there were fewer 
than three states per lab. 13 In FY66 $26.8 million (in constant 
1982-84 dollars) were allocated to the labs and doubled during the 
next year (see figure 2) . The total amount of money in constant 
dollars for the labs peaked in 1970 and the present total 
expenditures on those institutions is only slightly more than a 
third of what it was three decades ago. 



Insert figure 1 



Compared to the expectations of the Gardner Task Force, the 
labs have always been rather modest in size. The initial labs rose 
from an annual average of $1.4 million (in constant 1982-84 
dollars) in FY66 to an annual average of $4.3 million in FY70 — a 
level at which they uere maintained or even slightly increased 
despite dramatic reductions in the total number of labs in the 
early 1970s (see figure 2) . The average annual funding for labs 



Herd S. Ma.xm, "Regional Educational Laboratory Approaches 
to Educational Improvement: A Descriptive Synthesis, M Final Report 
of the Laboratory Synthesis Project, RFQ 108331 (December 1988), p. 
II-8. 
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declined in the second-half of 1970s and 1980s so that today the 
average annual lab budget in constant dollars is only $2.3 million- 
-about half of its earlier level. 



Insert figure 2 



There was also considerable debate over the national versus 

the regional orientation of the labs. The Gardner Task Force had 

emphasized the national nature of the labs, but many of the staff 

at the U.S. Office of Education (USOE) stressed their regional 

functions — in part because they felt that every congressperson 

wanted a lab in their own region. Francis Keppel, an early 

proponent of national rather than regional labs, switched his 

position. When someone challenged Keppel about his support of 

regional labs, he justified his new position because "Title IV labs 

are going to be a porkbarrel. Every Congressman is going to want 

one in his region." 14 Moreover, since the designation of 

"national" labs raised fears about excessive federal control , it 

15 

seemed more prudent to call them "regional" labs. This tension 
between the "national" and "regional" focus of the labs was not 
satisfactorily or conclusively settled and it even remains as an 
important unresolved issue today. 



9 
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14 Quoted in Richard A. Dershimer, The Federal Government and 
Educational R&D (Lexington, MA: Lexington Books, 1976), p. 86. 

15 For a thorough discussion of the early debates over the 
nature and function of the labs within the U.S. Office of Education 
(USOE) , see Dershimer, The Federal Government and Educational R&D , 
pp. 83-103. 
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The initial reactions to the labs were mixed. In a very 

influential and frequently cited analysis of the R&D centers and 

labs in the late 1960s, Francis Chase, Dean of the School of 

Education at the University of Chicago and a close ally of Gardner, 

enthusiastically endorsed the concept of labs as well as how they 

were being managed (though he also had some constructive 

16 

suggestions for how to improve their operation) . He saw the 

R&D centers and labs as complementary activities and cited several 

17 

instances of close co-operation between them. Chase 
particularly stressed the joint role of the centers and labs in 
systematically developing educational products and disseminating 
them to schools. 



While his public statements about the labs generally were 
laudatory, privately he seems to have been quite aware of the 
serious shortcomings in the functioning of many of the labs. For 
example, in a retrospective interview in the early 1970s, he 
observed that the labs "were going in all directions — service- 
oriented, research-oriented, giving grants to people who couldn't 
have gotten them in national competition, some behaving like state 
departments of education, others behaving like weak schools of 
education. A lot of trial and error." Quoted in Dershimer, The 
Federal Government and Educational R&D , p. 92. 

17 Francis S. Chase, "The National Program of Educational 
Laboratories: An Independent Appraisal of Twenty Educational 
Laboratories and Nine University Research and Development Centers 
Conducted Under Contract No. OEC-3-7-001536-1536" (Washington, DC: 
U.S. Department of Health, Education, and Welfare, 1968), pp. 9-12. 
For example, Chase pointed to the cooperation between the Research 
and Development Center at the University of Pittsburgh under the 
direction of Robert Glaser and Research for Better Schools, one of 
the local labs in developing and disseminating Individually 
Prescribed Instruction (IPI) . While there certainly were some 
instances of close center-lab cooperation, Chase greatly 
exaggerated their prevalence and seriously under-estimated the 
difficulties involved in getting R&D centers and labs to work 
together as a team. 
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Centers typically address themselves to identification 

and/or formulation of theoretical models or systems to 

perform stated functions; and sometimes to the design of 

components or elements required to move from conceptual 

to working models and assembly of components into 

consistent systems. Centers and laboratories share in 

testing working models or systems in laboratory 

situations to reveal malfunctioning, unsolved problems, 

and undesired side-effects; and in subsequent refinement 

and redesign as needed to correct defects and increase 

power and efficiency. The laboratories typically arrange 

further testing under a variety of field situations to 

gauge performance more precisely, to reveal modifications 

required by characteristics of the population served and 

other variations in situations. Collaborating centers 

and laboratories share responsibility for progressive 

precision in specification of intended effects and of the 

resources and processes necessary to produce the desired 

effects. Careful analysis of the yields or benefits of 

the new or revised system under specified conditions, and 

of measures and costs involved in maintaining the 

specified conditions is likewise a joint 

18 

responsibility. 



18 Chase, "The National Program of Educational Laboratories, 11 
pp. 26-27. 
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Chase wanted the laboratories to focus on a few large and 
long-term projects rather than spread their efforts and resources 
too broadly. 19 Moreover, while he acknowledged the regional 
nature of the labs, he saw them more as national institutions which 
would provide systematically developed and field-tested products 
well beyond their local areas. 20 He also recognized that any 
serious development efforts would be expensive and time-consuming: 



Chase felt that the centers were generally more focused in 
their activities than the labs. He applauded the recent efforts of 
the labs to redefine their missions more narrowly and to focus 
their attention on a smaller number of projects and activities. 
Chase, "The National Program of Educational Laboratories," pp. 18- 
21. The U.S. Office of Education also believed that the mission of 
the labs was to be focused. As a representative of that office put 
it, "The functions of these laboratories are to identify the major 
educational problems of the region, to choose one or two priority 
areas in which to mount an effective program, and to devise and 
administer a coordinated program to solve those problem areas. ... 
Regional laboratories will be doing research activities where these 
obviously bear on the problems attacked." U.S. Congress, House, 
Committee on Education and Labor. Study of the United States 
Office of Education . 90th Congress, 1st Session (House Document 
No. 193: Washington, DC: U.S. Government Printing Office, 1967), p. 
227. 

20 "Experience seems to indicate that the more successful 
laboratories achieve national visibility and influence because of 
the power of the concepts with which they are working and their 
ability to incorporate these concepts in operational systems of 
superior performance.... My conclusion is that it is desirable to 
have one or more laboratories in the Northeast, the Northwest, the 
Southeast, the Southwest, the North Central, the South Central, and 
other major regions; but this does not mean that there is any 
special validity in the present so-called regional grouping of 
laboratories." Chase, "The National Program of Educational 
Laboratories," pp. 36-37. While the tension between regional and 
national labs did not disappear, in the late 1960s and early 1970s, 
the U.S. Office of Education (OE) and its successor, NIE, stressed 
their national orientation. Testimony of Richard Rossmiller, 
U.S. Congress, House, Subcommittee on Select Education, Committee on 
Education and Labor, National Institute of Ed ucation: Hearings 
before the Subcommittee on Select Education . 94th Congress, 1st 
Session (Washington, DC: U.S. Government Printing Office, pp. 101- 
102. 



108 

Some of the curriculum studies supported by the 
National Science Foundation required expenditures of the 
order of a million dollars a year for several years; and 
while they produced notable improvements in texts and 
other instructional materials, none was subjected to the 
extensive development and testing which are necessary for 
reliable performance. While precise calculations are not 
yet possible, there is reason to believe that the 
development , production , testing, and refinement of even 
a fairly limited instructional system, requires a staff 
of fifteen or more highly qualified specialists and 
general ists with a supporting technical and clerical 
staff. Special facilities are also required for 
experimentation, design of prototypes, production of 
components, assembly of components into systems, and 
rigorous testing at each stage of development.... As the 
actual development of components and the fitting together 
of systems occurs, the annual cost for a single well 
designed program is likely to exceed a million dollars. 
If it is assumed that an organization will be engaged 
simultaneously in the development of three or more 
complementary programs or systems, the annual funding of 
the organization might easily exceed three million 
dollars. 

21 Chase, The National Program of Educational Laboratories," 
pp. 51-52. In constant 1982-84 dollars, the annual $1 million cost 
of each development project would be $2.9 million. Thus, the 
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While Chase whole-heartedly endorsed the concept and 
activities of the labs, other analysts were less sanguine about 
their achievements and direction. The U.S. House Special 
Subcommittee on Education, which was investigating the U.S. Office 
of Education, expressed strong reservations about the actual 
operation of the labs. For example, the Subcommittee found 
considerable confusion about the mission of the labs — even among 
some of the staff of the labs. Similarly, the Subcommittee often 
found no clear distinction between the R&D centers and the labs. 
Moreover , the members voiced grave concern about the large 
proportion of high salaries among lab personnel. However, both the 
Subcommittee and Chase did agree that the erratic and uncertain 
funding of the labs made -ny long-term planning extremely 
difficult. 22 

Just as it appeared that the situation of the labs in the late 

1960s finally might be stabilizing and improving, serious 

congressional opposition to an expansion of their funding arose. 

Some of this opposition was based on changes in key legislators or 

alterations in their responsibilities; but much of the opposition 

was due to the competition for scarce funds as a result of the 

23 

escalation of the Vietnam War. 

minimum size of a lab which had at least three development projects 
would be about $9 million (in constant 1982-84 dollars). 

U.S. Congress, House, Study of the United States Office of 
Education , pp. 224-229, 240. 

John Fogarty, chairman of the House Subcommittee on 
Appropriations for the Department of HEW and a staunch supporter of 
federal research died in 1968. Moreover, Congressman Edith Green, 
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The creation of 20 relatively small labs in the mid-1960s had 
been predicated upon the expectation of a major expansion of 
federal funding for them. While the White House did propose a 50 
percent increase in lab funding for FY69, the Congress provided 
only a 4 percent increment. As a result, the U.S. Office of 

Education (USOE) decided to terminate five of the labs and signaled 

24 

the end of the planned rapid expansion of the labs. Although 
there was a modest increase in lab funding in FY70, a 14 percent 
decrease the following year led to the closing of another four 
labs. The even larger declines in funding of the newly created 
National Institute of Education (NIE) meant continued reductions in 

federal support for laboratories and by 1975 only eight labs 

• * 25 
survived. 

At the same time that the overall budget of the labs was being 
reduced in the early 1970s, the manner in which the labs and the 
R&D centers were funded was also being reviewed and revised. The 
Gardner Task Force and the report by Francis Chase had argued for 
large-scale labs with stable funding. But the attacks upon the 

chairperson of the Special Subcommittee to Study the Office of 
Education, began to attack the labs — especially since she thought 
that the lab staffs were overpaid. But the budget constraints due 
to the increased costs of the Vietnam War created a climate in 
which' all domestic spending increases became more difficult. 
Dershimer, The Federal Government and Educational R&D , pp. 95-98. 

24 Dershimer, The Federal Government and Educational R&D , pp. 
98-101. 

25 For an excellent discussion of the problems of NIE and its 
funding, see Lee Sproull, Stephen Weiner, and David Wolf, 
Organizing an Anarchvt Belief. Bureaucracy, and Politics in the 
National Institute of Education (Chicago: University of Chicago 
Press, 1978) . 



Bureau of Research within the u.S, Office of Education by the 
Congress and other agencies within the Administration for failing 
to provide any immediately useable products led to a fundament?! 
redefinition of the research and development process. No longer 
content to let individual researchers or institutions pursue their 
own personal agendas , the Bureau of Research stressed "mission- 
oriented" research — a concept borrowed from the Defense Department. 
Labs and R&D centers now also were expected to compete directly for 
specific research projects rather than receiving their federal 
funds outright. This "program purchase" approach initiated by the 
Bureau of Research was continued when labs and R&D centers were 
transferred to the National Institute for Education (NIE) (created 
in June 1972) . 26 

♦ 

The labs and R&D centers strenuously objected to the program 
purchase policy because it fragmented their activities and 
emphasized short-term planning. They had banded together for 
informational purposes in 1969 at the insistence of the federal 
government. While initially lobbying was not a major focus , the 
Council for Educational Development and Research (CEDaR) moved to 

Washington , DC in 1974 and argued for the Ford Administration and 

27 

the Congress to reverse the program purchase policy. 
2 6 

Samuel D. Sieber, "Federal Support for Research and 
Development in Education and its Effects," in The S even t y- th i rd 
Yearbook of the National Society for the Study of Education , C. 
Wayne Gordon, ed. , Part 2 (Chicago: University of Chicago Press, 
1974), pp. 478-502; Spoull, Weiner, & Wolf, Organizing an Anarchy . 

27 

Initially, CEDaR had not opposed the program purchase 
policy, but only did so when it became evident that it would 
fragment tneir institutions. Carolyn Breedlove at the National 
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NIE and the National Council on Education Research (NCER) 

commissioned ten consultants to do a quick three-month review of 

the funding policies of NIE. Most of the consultants were 

prominent educators and many had considerable previous experience 

with research and evaluation activities. While the consultants 

raised important questions about the effectiveness of the present 

lab system, they observed that "for us the only question about the 

basic concept of the laboratories is how to make it work well, not 

whether the laboratories should exist. The need for established, 

28 

long-term, R&D institutions still impress us." 

The consultants made eight important recommendations which 
reflect their particular vision for the role of the labs in the 
mid-1970s. Since their suggestions build upon the earlier 
expectations for the labs and differ considerably from many of the 
assumptions today, it may be worthwhile to quote them at some 
length: 

First, there should be a small number of very high 
quality institutions, perhaps no more than a half 
dozen • . • . 

Second, each institution should center on a mission, 
closely related to a priority of the major sponsoring 

Education Association (NEA) is now completing an indepth analysis 
of the origins and evolution of CEDaR. 

28 Ronald F. Campbell et al., R&D Funding Poli cies of the 
National Institute of Education: Review and Recommendations . 
National Institute of Education (Washington, DC: Department of 
Health, Education, and Welfare, 1975), p. 24. 

erJc 12: 
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agency, the National Institute of Education. The work of 
the organization would be to take ideas from their 
inception through their development, refinement, and 
testing where appropriate, to dissemination.... 

Third, the organizations 1 funding. . .must be stable 
(three to five years) , it must come chiefly from one 
source (NIE) , and it must be clearly tied to specific 
work agreed upon in advance, and it must be in the 
magnitude of $3 to $4 million per year at least.... 

Fourth f the organ i z at i on must be protected f rom 
demands to give undue services to local and state 
agencies unrelated to the major R&D mission... a [W]e are 
cautioning against seeing a small number of national 
laboratories as places where practitioners might go and 
expect advice on education problems generally. 

Fifth, it should be clear that the unusual guarantee 
of continued funding will bring with it a need for an 
unusual degree of monitoring and review of the work. . . . 

Sixth, the pursuit of other funds by the 
organization should be subject to review and perhaps 
limitation, in order to insure the focus on the major 
goals of the chief sponsoring agency, NIE.... 

Seventh, the redesigned laboratories would no longer 
be designated "regional", though we feel they should be 
located in different parts of the country. So long as 
they expect substantial Federal support in an era of very 
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tight Federal R&D dollars, they must be working on parts 
of the national R&D agenda, as set through a national 
process, . . • 

Eighth, lastly, we are concerned that as such strong 
and unique institutions mature further, they maintain a 
prime commitment to effecting change and improvement in 
schools...* No amount of sophistication in the R&D work 
can compensate for irrelevance to the world of 

educational practice, and the renewed laboratories should 

29 

not forget this. . . . 

Since this analysis of the labs and R&D centers was produced 

amidst substantial decreases in NIE funding, there was concern that 

a proper balance of research activities be maintained. Therefore 

the consultants recommended that "no more than about a third of 

NIE 1 s program funds be allocated to work at the resulting special 

30 

institutions [labs and R&D centers]." 

Sam Sieber, one of the ten consultants, also added an appendix 
to this report which detailed his views on the requirements of a 
national educational R&D system. He addressed a topic which had 
not been widely discussed in the other reports — the importance of 



29 Campbell et al., R&D Funding Policies of the National 
Institute of Education , pp. 25-27. 

30 Campbell et al. , R&D Funding Policies of the National 
Institute of Education , p. 72. 
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"excellence or quality control." Sieber briefly noted some of 
the factors which have led to the neglect of concern about quality- 
-such as "the vagueness of evaluative criteria , the lack of 
consensus on procedures , and the failure of a major sponsor — NIE — 
to develop any agency-wide mechanism for assessing quality He 
went on to recommend that "[p]erhaps what is vitally needed is a 
national task force or commission on the quality of educational 
R&D." 32 Unfortunately, his plea for examining the quality of the 
work produced by NIE or later OERI has been ignored almost 
entirely. 

Not everyone agreed with all of the recommendations of the 
Campbell panel — particularly those suggestions which affected the 
labs. Richard Rossmiller, Chairman of CEDaR, and Robert Scanlon, 
member of the board of trustees of CEDaR, appeared at the House 
Hearings on NIE reauthorization and questioned wisdom of the 
proposed seemingly exclusive national orientation of the labs: 

With all due respect to the consultants, such an 



9 
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Campbell et al., R&D Funding Policies of the National 
Institute of Education , p. 90. Sieber illustrated the dangers of 
not worrying about the quality of the research being disseminated. 
"The extent to which faddism dictates the adoption of innovations 
of doubtful merit has been demonstrated in a recent study of 
virtually all big-city secondary schools. This study reveals that 
while 46 percent of these schools are relatively high in adoptions 
of innovations, most of the innovations adopted by half of these 
innovative schools are of relatively low quality (as judged by a 
national panel of secondary school experts) . Clearly, quality 
remains as important an issue as quantity of adoptions. 

32 Campbell et al., R&D Funding Policies of the National 
Institute of Education , p. 90. 
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attitude smacks of elitism. Even more important, it reflects 
a lack of understanding about how effective r & d institutions 
relate to the field. The Campbell panel says that these 
national laboratories should do extensive field testing in 
schools. Well, we can guess what the attitude of the school 
practitioners would be to our request to bring our prototype 

products into their schools after we've turned down their 

33 

request for assistance to a problem. 



Rossmiller and Scanlon did not object altogether to a national 
orientation for the labs, but only to an exclusive one at the 
expense of their regional ties. "In other words, the labs and 

centers are "national 1 in terms of their scope of work. But they 

34 

should also maintain strong ties to their region and state." 

The issue of the governance of the labs was also raised. The 
Campbell Report suggested that NIE should take a larger role in 
setting the priorities of the labs as well as in monitoring them 
more closely. Rossmiller and Scanlon disagreed with the panel "s 
recommendation : 
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U.S. House , Subcommittee on Select Education , National 
Institute of Educations Hearings . . . 1975 , p. 109. Richard 
Rossmiller also was the Director of the Wisconsin Research and 
Development Center for Cognitive Learning and Richard Scanlon was 
Executive Director, Research for Better Schools, Inc. , 
Philadelphia, PA. 

34 U.S. House, Subcommittee on Select Education, National 
Institute of Education: Hearings. 1975 . p. 110. 
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We be 1 ieve such a recommendat ion has inherent danger , 
The panel is recommending that the federal government manage 
the operation of the laboratory. The influence of the 
practitioner, in other words, would be negligible. The 
laboratory would simply be an extension of the Institute, an 
ivory-tower think tank isolated from the practitioner it's 
intended to serve. Furthermore, the issue of federal control 

over the development of a "national" curriculum would 

35 

certainly be raised somewhere. 

But Rossmiller and Scanlon did agree with several of the other 
recommendations of the Campbell panel such as the need to continue 
support of NIE in general and of the labs and centers in 
particular. They also called for a rigorous evaluation of all NIE 
products — including those produced by the labs and centers. 

The questions, "How well does it work?" and "what 
difference does it make?" are frequently raised about the 
outcomes of research and development. The laboratories 
and centers take the position that quality control in 
product development is the single most important variable 
in their work. However, government pressure to 
disseminate products, coupled with the expense of 
ensuring quality control, often works against us. We 

35 U.S. House, Subcommittee on Select Education, National 
Institute of Education: Hearings. .. 1975 , p. 110. 
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believe that NIE has the responsibility for developing, 
in conjunction with educational practitioners and the 
research and development specialists, effective quality- 
control procedures. 36 

The National Council on Educational Research (NCER) , the 
policy advisory group to NIE, "reviewed the consultants 1 report and 
is in general agreement with both their analysis and conclusions, " 
It agreed that NIE "should take responsibility for the general 
institutional health of educational research and development and 
that substantial Institute resources should be directed toward a 

group of research and development institutions working directly 

37 

with NIE on a long-term, large-scale basis," The Council 

called for the establishment of two to four national laboratories 

3 8 

selected from among the existing R&D centers and labs. 
Moreover, as the "new system is gradually phased in, NIE should 
continue to reserve a substantial (but declining) proportion of its 
budget for supporting, developing, and strengthening existing 



U.S. House , Subcommittee on Select Education , National 
Institute of Education; Hearings . .. 1975 , p. 116. 

37 National Council on Educational Research, "Resolution of 
the National Council on Educational Research Institutions Engaged 
in Education Research and Development" (NCER Resolution No. 091875- 
1 , July 23, 1976) , p. 1. 

38 U.S. House, Subcommittee on Select Education, National 
Institute of Education: Hearings. .. 1975 , p. 111. 
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39 

laboratories and centers capable of high quality research." In 

an appendix to its document, the Council detailed its vision of the 

new national labs and the old regional labs. The regional labs 

were to continue to provide assistance and services to 

practitioners in their local areas while also conducting R&D 

40 

programs of national significance. Thus, the Council opted for 
a middle course between the almost exclusive national orientation 
for the labs according to the Campbell panel's recommendations and 
the desire of others to maintain a regional focus for the existing 
labs. 

Five key members of the U.S. Senate Committee on Labor and 

Public Welfare strongly objected to the efforts of NIE and NCER to 

create the proposed new national labs while at the same time 

gradually reducing the number or the funding of the existing 
4 1 

regional labs. In response, Harold Hodgkinson, the Director of 
NIE, quickly reassured everyone that no attempt would be made to 
revise dramatically the lab system or to phase out existing labs as 
long as they fulfilled their stated missions. "The question, then, 
is not whether we will support labs and centers, but how we will 
insure that a lab and center program and the institutions in it are 



National Council on Educational Research, "Resolution of 
the National Council on Education Research Institutions Engaged in 
Education Research and Development," p. 1. 

40 National Council on Educational Research, "Resolution of 
the National Council on Education Research Institutions Engaged in 
Education Research and Development," Appendix C. 

41 Jennings Randolph et al, "Letter to Harold L. Hodgkinson," 
(January 27, 1977) . 
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as strong as possible." Hodgkinson went on to reiterate that 

federal responsibility for the labs and centers should include a 

"rigorous assessment of these institutions, .. and, if necessary, [a] 

phase out of special relationships with those which can't pass 

42 

muster after considerable support from NIE. M 

While NIE Director Hodgkinson readily acquiesced to the 
congressional pressures , the National Council on Educational 
Research (NCER) was more hesitant. NCER withdrew its original 
proposal to create a few national labs and to reduce the number of 
regional labs, but they chastised Hodgkinson for agreeing to a 
"special institutional relationship" with the labs before the 
quality of each of their institutions had been established. 
Indeed, a NCER Committee to review these issues felt "that the 
Institute is not paying sufficient attention to considerations of 
quality and relevance which the Committee believes to be at least 
equal in importance to considerations of stability and political 
comity." The NCER Committee went on to define quality in a broad, 
comparative sense (and one that might be worth resurrecting even 



Harold L. Hodgkinson, "The Official NIE View of Labs and 
Centers," National Institute of Education Memo (March 1, 1977), p. 
2. Hodgkinson acknowledged the previous hostility between NIE and 
the centers and labs and admonished the NIE staff to change their 
behavior. "We all know that the history of the NIE/lab and center 
relationship has been fraught with distrust so it is not surprising 
that our lab and center policies are received with some skepticism. 
What concerns me, however, is that we continue to fuel this 
paranoia by informally conveying anti-lab and center sentiments in 
our conversations and behavior. This behavior and the image it 
projects proves detrimental to the Institute since much of our 
difficulty with Congress and the labs and centers is due to the 
appearance and not the reality of our lab and center policies." 
Ibid, p. 1. 
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A principal test of quality and merit , however , must be 

a comparison with other potential awardees. This is true of 

all Federal agencies engaged in fields where absolute tests 

are not available. It is not clear that this emphasis on 

quality , merits and comparative judgment is being adequately 

emphasized in the NIE review process. Too often , when 

considering labs and centers in relation to other R&D 

organizations , the Institute finds itself in the posture of 

having to prove that a lab or center is incompetent or 

mediocre in order to shift funding from previously-established 
43 

patterns. 

If the Campbell report of 1975 reiterated the Gardner Task 
Force 1 s vision of large-scale national labs, directives from the 
Congress led to a rather different view of the labs. First, the 
Congress legislatively mandated the continued existence of the labs 
and R&D centers. Second , it called for the creation of a new panel 
to review and assess the existing labs and R&D centers and to make 
recommendations for their long-term development. As we shall see, 
this important panel rejected the Gardner Task Force national 
vision and moved toward more regional labs — a position compatible 
with the growing sentiment within segments of NIE itself which 



43 John E. Corbally, "Review of Resolution 18," National 
Council on Education Research memo (May 19, 1977), pp. 6-7. 
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despaired that the existing labs were neither producing high 
quality products nor responding adequately to suggestions from the 
NIE lab monitors . 

Under congressional pressure for the more immediate impact of 
research and development upon schools and their call for more 
dissemination, the new Director of NIE, Patricia Graham, tried to 
involve more educational practitioners in NIE activities. Based 
upon earlier negotiations between NIE and CEDaR, among the 15 
individuals for the congress ionally-mandated panel to review the 
labs and R&D centers, Graham selected a sizable contingent of 
practitioners. 44 The placement of a large number of 

practitioners on the new panel was a major departure from the 
composition of the Campbell group and probably contributed to the 
redirection of the mission of the labs. As the Panel for the 
Review of Laboratory and Center Operations remained in operation 
for two years and was active throughout the entire period, the 
group had considerable impact on how labs were perceived and 
treated. 

The Panel for the Review of Laboratory and Center Operations 
rejected the national orientation proposed by the Campbell group 
and recommended more regionally controlled and oriented labs. They 
saw labs as designed to: 

44 NIE and CEDaR had reached an agreement in a meeting on 
September 3, 1976 that M no less than half of the Panel members will 
be educational practitioners (with an emphasis on elementary and 
secondary practitioners)." Harold L. Hodgkinson and Richard A. 
Rossmiller, "Memorandum of Agreement Between NIE and Labs and 
Centers from the September 3 Meeting, " National Institute of 
Education Memo (October 4, 1976). 
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identify concerns and priorities through regionally 
representative governing and advisory structures and 
activities that help the regional clientele define their 
needs ; 

conduct applied research and development in pursuit of 
those priorities; 

provide technical assistance to the region; 

facilitate communication among agencies and individuals; 

promote the use in the region of R&D results from all 
sources ; and 

disseminate the results of their own R&D on a national 

. . 45 
basis. 

While the panel accepted research and development as 
legitimate lab activities , it stressed the regional nature of these 
activities. Moreover, the panel emphasized the role of labs in 
providing technical assistance to the regions and stressed the 
importance of dissemination. Thus, the initial vision of the labs 
as large-scale national research and development institutions 
proposed by the Gardner Task Force and reiterated by Francis Chase 
and the Campbell panel in their investigations was rejected and 
thereafter neither NIE or OERI again put forth this view as an 
ideal model for the labs. 

45 

Panel for the Review of Laboratory and Center Operations, 
Research and Development Centers and Regional Educational 
Laboratories; Strengthening and Stabilizing a National Resource . 
Final Report, National Institute of Education (Washington, DC: U.S. 
Department of Education, 1979), p. 8. 
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Another major change for the labs in the late 1970s was NIE's 

growing hostility to supporting the large-scale development of 

curriculum. The federal government became increasingly involved in 

supporting curriculum development after the Russians launched 

Sputnik in 1957. 46 The National Science Foundation (NSF) spent 

47 

$180 million from 1957 to 1975 to support curriculum projects. 

And the Gardner Task Force in 1964 had explicitly recommended that 

48 

the labs "would develop and disseminate ideas and programs." 

As a result, many of the R&D centers and labs in the late 1960s and 

early 1970s developed and distributed curriculum packages. The 

University of Wisconsin R&D Center created the Individually Guided 

Education (IGE) project and CEMREL, the St. Louis based lab, 

49 

developed the "Comprehensive School Mathematics Program." 

As the federal agencies entered the area of curriculum 
development, they increasingly faced critics who argued that this 



46 

For an analysis of curriculum changes in the past, see 
Larry Cuban, "Determinants of Curriculum Change and Stability, 
1870-1970," in Value Conflicts and Curriculum Issues; Lessons from 
R esearch and Experience , eds. Jon Schaffarzick and Gary Sykes 
(Berkeley, CA: McCutchan, 1979), pp. 139-196; Herbert M. Kliebard, 
"Systematic Curriculum Development, 1890-1959," in Value Conflicts 
and Curriculum Issues , eds. Schaffarzick and Sykes, pp. 197-236. 

47 J. Galen Saylor, Who Planned the Curriculum? A Curriculum 
Plans Reservoir Model with Historical Examples (West Lafayette, IN: 
Kappa Delta Pi Press, 1982), p. 64. 

48 Gardner , "Report of the President 1 s Task Force on 
Education," p. 34. 

49 Saylor, Who Planned the Curriculum? pp. 66-67. For a 
recent discussion of some of these efforts, see Richard C. Atkinson 
and Gregg B. Jackson, eds., Research and Reform: Roles for th e 
Office of Educational Research and Improvement (VJashington, DC: 
National Academy Press, 1992), pp. 25-45. 
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was an inappropriate role for the national government. Particular 
criticism was directed at the NSF-funded upper elementary school 

social studies course, "Man: A Course of Study" (MACOS) which was 

50 

accused of subverting American traditional values and beliefs. 
Moreover, some publishers attacked the courses developed by the R&D 
centers and the labs for duplicating and undercutting their 
efforts. As a result, the National Council on Educational Research 
(NCER) appointed a Task Force to examine and guide NIE's curriculum 
development activities. 

The NIE Curriculum Development Task Force examined the issue 
in considerable detail and proposed four policy alternatives 
ranging from having NIE totally abandon any curriculum development 
activities to supporting the existing freedom for R&D centers and 
labs to undertake full-scale curriculum development. NIE Director 
Hodgkinson announced in early 1977 that: 

NIE's primary contribution to the improvement of instructional 
programs and materials is to sponsor (1) the conduct, 
synthesis, and dissemination of relevant research, (2) efforts 
to strengthen, facilitate, and coordinate others 1 work in 
improving instructional programs and materials, and (3) the 
prototypic development of new instructional programs and 
materials. Among these, research activities represent the 

50 Jon Schaf farzick, "Federal Curriculum Reform: A Crucible 
for Value Conflict," in Value Conflicts and Curriculum Issues , eds. 
Schaf farzick and Sykes, pp. 1-24. For an indepth analysis of the 
MACOS project, see Peter B. Dow, Schoolhouse Politics: Lessons from 
the Sputnik Era (Cambridge, MA: Harvard University Press, 1991). 
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Institute 1 s highest priority, followed by efforts to 

strengthen, facilitate, or coordinate others 1 work, and then 

prototypic development. Finally, full-scale development may 

be considered by NIE, but only for certain limited purposes 

51 

and only when none of the primary activities will suffice. 

NIE's limited support of curriculum development activities as 
well as the mandate from Congress for the Institute not to be 
involved in this area had a profound effect on many of the R&D 
centers and labs. As the R&D centers abandoned development, most 
of them simply turned their attention to other areas of research. 
For the labs, however, the decision to abandon NIE-funded full- 
scale curriculum development meant that much of their previous 
extensive research and evaluation efforts were now abandoned 
altogether. Coupled with the growing focus on regional issues and 
the provision of technical assistance, many of the labs gradually 
lost the high quality personnel who had been working on curriculum 
development and who were an indispensable component of the Gardner 
Task Force's original vision for staffing of the labs. 

Another major and at first seemingly cataclysmic event for the 
labs was the presidential election of 1980. Although partisan 



51 Harold L. Hodgkinson, "NIE's Role in Curriculum 
Development : Findings , Pol icy Options , and Recommendations , " 
National Institute of Education, February 8, 1977 [prepared by Jon 
Schaffarzick and Gary Sykes] , p. 109. The definition of 
"prototypic development" they used was the "design and production 
of an exemplary part of an instructional program (e.g., a one-week 
unit in a two-semester course) to serve as a model for continued 
development . " Ibid . , p . 3 . 
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changes in the past did not necessarily seem to have much direct 

impact on the running of the labs, the 1980 election brought to 

power individuals who no longer accepted some of the basic 

assumptions of those who had staffed NIE during the Nixon , Ford, 

and Carter Administrations. Just as NIE and CEDaR were beginning 

to reach a mutual accommodation in the late-1970s on how to view 

and manage the labs, Edward Curran, the first Reagan appointee to 

head NIE, sought to abolish the agency entirely. Moreover, Curran 

and his successor, Robert Sweet, dismissed many of the NIE 

employees and replaced them with individuals who were skeptical of 

the existing education establishment. While Curran and Sweet did 

not succeed in eliminating NIE, thanks in part to the strong 

interventions of the Secretary of Education, Terrel Bell, the 

turmoil they brought to NIE disrupted many of the working 

relationships which had begun to crystalize earlier — including the 

52 

emerging partnership between CEDaR and NIE. 

The difficulties at NIE were compounded by the substantial 
reductions in the budget as part of the overall effort to reduce 
federal domestic spending. NIE ■ s budget was reduced from $82 
million in FY80 to $53.2 million in FY82 — a 35 percent decrease. 
If the lab and center funding were to remain constant, as had been 
previously negotiated as part of their five-year contracts, little 
money would have been left over for other projects in FY82. The 

52 

For a detailed analysis of the changes and turmoil at NIE 
between 1981 and 1983, see Philip Phaedon Zodhiates, "Bureaucrats 
and Politicians: The National Institute of Education and 
Educational Research Under Reagan," Unpub. Ed. D. diss., Harvard 
University , 1988 . 
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Omnibus Reconciliation Act of 1981 stipulated that NIE could 
proportionately adjust downward all of its grants and contracts to 
finance the decrease, but CEDaR successfully lobbied to protect 
most of the lab and center funding. The Continuing Resolution 
Appropriation specified that lab and center funding could be cut by 
no more than 10 percent for FY82. NIE did reduce lab and center 
funding by the full 10 percent, but the magnitude of the overall 
budget reduction meant that other NIE programs had to be cut even 
more severely. 

There had been periodic complaints that while the labs and 
centers continued to receive a disproportionate share of NIE funds, 
they did not have to face any open competition to maintain their 
funding. Therefore, the Omnibus Reconciliation Act of 1981 also 
stated that labs and centers. ... "shall upon completion of existing 
contracts, receive future funding in accordance with government- 
wide competitive bidding procedures and in accordance with 
principles of peer review involving scholars and State and local 
educators to ensure the quality and relevance of the work 
proposed. 1,53 

Curran and his successor Sweet wanted to terminate the 
existing lab and center contracts immediately in 1982 — arguing that 
this would have honored the initial three-year agreements with 
these institutions. But the labs and centers objected and insisted 
that their entire five-year contracts or grants with NIE be funded. 

53 U.S. House of Representatives, Omnibus Budget 
Reconciliation Act of 1981: Conference Report (H.R. 3982, Report 
No. 97-208, 1981), p. 729. 
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CEDaR succeeded in getting the Senate Appropriations Committee to 

54 

require NIE to fund these institutions through 1984. 

Manuel Justiz, who succeeded Sweet as the Director of NIE, 
initiated a highly public and open competition for the labs and 
centers. Included in this process was an extensive examination of 
the purposes and functions of the laboratories which culminated in 
a set of recommendations by the NIE Laboratory Study Group. The 
Laboratory Study Group listed a variety of tasks for these 
institutions, but noted that "[laboratory responsibility would be 
relatively less in the area of research and relatively greater in 
development and other transformations of research to directly 
useful products and processes." 55 The Study Group also 



considered five more specialized functions for the labs, but 

concluded that the existing "general purpose" laboratory model with 

56 

its multiple constituencies and multiple purposes was best. 
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54 Zodhiates, "Bureaucrats and Politicians," pp. 85-90. 

55 NIE Laboratory Study Group, "Expanding and Strengthening 
NIE's Regional Laboratory Services: Needs, Issues, and Options," 
(Washington, DC: National Institute of Education, October 3, 1983) 9 
p. 3 . 

56 NIE Laboratory Study Group, "Expanding and Strengthening 
NIE 1 s Regional Laboratory Services, " p. 10 . The five options 
considered and rejected were "(1) laboratories focusing on research 
and technical assistance for education policymakers in state and 
local government, including and [sic] boards of education; (2) 
laboratories working with a broad array of assisting organizations 
to improve local district, school, and classroom practice; (3) 
laboratories focusing on improvement of professional training and 
inservice development of teachers and administrators ; ( 4 ) 
laboratories providing direct service to local schools; and 
laboratories specializing in various substantive areas (e.g., 
reading math, science, technology, finance)." Ibid., 10. While 
the Study Group discussed the first three options in some detail 
and felt they deserved further consideration, they rejected options 

13C 



130 

The Study Group accepted the regional nature of the labs, but 

called for closer co-ordination between NIE's national priorities 

and the activities of the labs. They suggested a 50 percent 

increase in overall funding for NIE with a proportionate boost for 

the labs. "As additional funds are made available to NIE," the 

Study Group recommended "that NIE provide some resources for 

competitive opportunities for the laboratories, consistent with 

their basic purposes." Unlike the Campbell Panel in 1975 which 

worried about too much extraneous outside funding for the labs, the 

Study Group "recognized the importance of laboratories seeking 

other funding sources to expand the impact of their core NIE 

support." 58 The Study Group also called for more effective 

monitoring of the labs and urged a clearer distinction in functions 

between the labs and centers. Finally, they emphasized the need 

for closer co-operation between the labs and centers and stated 

that "NIE and the labs should interactively develop plans for 

59 

addressing national priorities." 

4 and 5 as inappropriate models for NIE's laboratories. 

57 NIE Laboratory Study Group, "Expanding and Strengthening 
NIE • s Regional Laboratory Services," p. 4. 

58 NIE Laboratory Study Group, "Expanding and Strengthening 
NIE • s Regional Laboratory Services," p. 4. 

59 NIE Laboratory Study Group, "Expanding and Strengthening 
NIE • s Regional Laboratory Services," p. 8. Whereas many of the 
plans for the laboratories in the late-1970s and early 1980s 
stressed the importance of the regional laboratory governing boards 
in setting the agenda, this report tried to balance national and 
regional interests and emphasized the need to arrive at a 
negotiated settlement. "NIE should establish a five-year R&D 
agenda focused on the remediation of identified weaknesses in 
American education. In response to this agenda, the lab and center 
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There was considerable skepticism that an impartial and open 

competition for the labs and centers would be held in 1985 . 

However, almost everyone agrees that NIE managed to conduct a fair 

60 

and effective peer review process. The Request for Proposals 
(RFP) for the laboratories identified five tasks for these 
institutions : (1) develop effective governance , management , 
planning and evaluation systems for the laboratory; (2) work with 
and through existing organizations to improve schools and 
classrooms; (3) work with State-level decisionmakers on school 
improvement issues; (4) work to create research and development 
based resources for school improvement; and (5) work in 
collaboration with centers and with other laboratories on regional 
and national educational problems. 61 

Many of the general trends in the treatment of the labs in the 
late 1970s and early 1980s were continued in the 1985 Request for 
Proposals. The regional orientation and control of the labs were 
maintained. Stress was placed on labs to provide technical 
services for their regional clients. 

But there were also some shifts in emphasis. In the past labs 



directors should submit to NIE tentative plans for their 
activities. After consultation with staff , the NIE Director should 
meet with each lab or center director to modify in a mutually 
acceptable fashion these long-range plans." Ibid . 

For an analysis of the peer review process in the 1985 lab 
and center competitions, see Thomas W. Schultz, "Behind Closed 
Doors: Peer Review in the NIE Research Center Competition, 11 Unpub. 
Ed. D. diss., Harvard University, 1988. 

National Institute for Education, "Regional Educational 
Laboratory Institutional Operations: Request for Proposal, 1985," 
(Washington, DC: National Institute of Education, 1984), pp. 18-25. 
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pursued a wide variety of tasks in furthering general educational 
improvement. Now labs were told to " focus on school and classroom 
improvement . H Moreover, labs were expected to " feature 
dissemination and assistance strategies . 11 Finally, labs not only 
were encouraged to collaborate with other labs and centers, but 

they were required to set aside a small, but fixed proportion of 

6 2 

their funds for this effort. 
♦ 

Research and development continued to be designated as lab 

activities, but now the stress was on short-term investigations, 

applied research and development, and the dissemination of 

research-based information to schools. Compared to the original 

vision of the labs in the 1960s and early 1970s, however, research 

and development activities were de-emphasized in the 1985 

competition. Only about 20-35 percent of the lab f s work program 

were assigned " to create research and development based resources 

6 3 

for school improvement . 11 

The entire concept of development had been substantially 
revised since the early 1970s. Whereas many labs had originally 
developed large-scale, field-tested curriculums, the guidelines now 
called for more modest products "such as research-based training 
designs, directories, guides or other practical materials that 



National Institute of Education, "Regional Educational 
Laboratory Institutional Operations," 1985, pp. 10, 11, 16. 

63 National Institute of Education, "Regional Educational 
Laboratory Institutional Operations," pp. 13-14, 25. 



ERIC 



14 



133 

64 

support the improvement process in their region." While NIE 
had been prepared at least to entertain the idea of some ambitious 
curriculum proposals earlier, now the Institute explicitly and 
unequivocally stated that "laboratories may not use NIE funds to 
engage in long-term curriculum development efforts." 65 

By the end of the 1985 competition, 9 labs were in operation — 
6 former ones and 3 new ones. 66 Labs were apportioned regionally 
with no overlap in jurisdiction. And the Office of Educational 
Research and Improvement (OERI) was reorganized so that the labs 
were placed within the Program for the Improvement of Practice 
(PIP) while the centers were overseen by the Office of Research 



National Institute of Education, "Regional Educational 
Laboratory Institutional Operations," p. 25. 

65 National Institute of Education, "Regional Educational 
Laboratory Institutional Operations, 1985," p. 14. NIE|s 
opposition to any large-scale curriculum development stemmed in 
part from the resurgence of hostility in the early 1980s to any 
federal involvement in this area. For example, at the public 
hearing on a national competition for the labs and centers in 
Chicago in 1983, Louise Kaegi a former teacher, indirectly 
challenged federal support for curriculum development by 
questioning the usefulness of values clarification and affective 
education in the schools. Similarly, Reverend Hiram Crawford 
stated that "many teachers are alcoholics, dope addicts, sex 
symbols and engage in teaching value clarification and secular 
humanism, whose basic philosophy is communistic. 11 National 
Institute of Education, "Transcript of Hearings on a National 
Competition for Regional Educational Laboratories and R&D Centers," 
Chicago, June 20, 1983, p. 177. Even more direct and vociferous 
attacks on federal support of curriculum development came at the 
Kansas City hearings where several participants questioned the NIE- 
supported work at CEMREL — one of the leading labs in curriculum 
development in the 1970s. National Institute of Education, 
"Transcript of Hearings on a National Competition for Regional 
Educational Laboratories and R&D Centers," Kansas City, June 22, 
1983. 



Due to congressional mandate, the North Central lab had 
been competed in 1984 rather than 1985. 
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(OR) . Moreover, a team of staff members were assembled in PIP to 
oversee the entire lab program and a particular staff person was 
assigned to each lab for monitoring. 

Chester Finn, the Assistant Secretary for OERI, launched an 
extensive external review of the labs in the summer of 1987. 
Christopher Cross, the former ranking minority staff member on the 
U.S. House Subcommittee on Select Education, headed up the outside 
review panel. Teams of external reviewers and an OERI lab monitor 
visited each of these institutions for two and one-half days and 
evaluated their progress and proposed 3-5 year plans (using 
standardized evaluation criteria that had been developed for this 
review) . 

The Cross Lab Review Panel concluded that PIP's external 
review process was thorough and competent, but they observed that 
,f [t]he panel received relatively little information about the 

67 

quality or impact of lab products." They went on to recommend 
that additional information about the quality and impact of the 
work should be obtained in the future. 

The Panel questioned the overall clarity and vision of the 
mission of the labs and noted "that the labs are very strongly 
oriented to their regions." While the Panel praised the 
regionality of the labs, they also felt that "there are some 
legitimate roles outside the region which labs might become 



Christopher T. Cross et al., "Report of the Laboratory 
Review Panel on the 1987 Review of Laboratories," Office of 
Educational Research and Improvement (October 6, 1967), p. 3. 
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involved with." 68 

The continued shift away from research at the labs as a result 
of the 1985 competition was noted and concern was raised about the 
availability of practitioner-oriented research: 

The 1985 recompetition resulted in a transition of labs from 

institutions which conducted some significant R&D on their 

own, to ones providing assistance services, primarily in 

partnerships with others. While the change has clear 

benefits, one cost is the loss of practitioner-oriented R&D 

that labs used to conduct. In part, this transition makes the 

choice of the R&D that labs incorporate in their services more 

critical. Based on knowledge presently available to it, the 

panel is not sure that there is a sufficient locus of 

practitioner-oriented research emanating from other sources 

69 

which the labs may draw upon. 

The Panel did not attempt to make any judgments on the overall 
quality of the labs, but did comment that since each of the 
external review teams looked only at one lab, they did not have an 
opportunity to make any comparisons. As a result, "the panel 
believes there is a tendency for reviews of this type to produce 
positive results.... This does not mean the results from this 
review are to be disbelieved, but the tendency for positive 

68 Cross, "Report of the Laboratory Review Panel," p. 7. 

69 Cross, "Report of the Laboratory Review Panel," p. 6. 
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70 

findings to result from such a process should be kept in mind." 

While the Cross Laboratory Review Panel hesitated in coming to 

any conclusions about the overall quality of the labs, Assistant 

Secretary Finn did not. After reviewing carefully the reports of 

the external review panels, the lab monitors, and the response of 

the labs, Finn expressed his strong doubts about the work of the 

labs in an internal OERI memo. Based upon this extensive two-year 

review, Finn concluded that he did not "see any evidence that the 

taxpayers' substantial investment in labs these past two years has 

71 

yielded any RESULTS of any sort." 

Assistant Secretary Finn frequently complained about the lack 
of flexibility in OERI research funding — due mainly to the 
congressional stipulation that the labs and centers receive the 
bulk of funds so that almost nothing was left for field-initiated 
research projects. While he had some criticisms of the centers, he 
reserved his strongest attacks for the labs. At a congressional 
oversight hearing on the functioning of OERI, Finn testified that: 

The laboratories, in particular, have not been a very 
renumerative investment per se . This is not to say that they 
do nothing useful — they and their energetic Washington 
lobbyists are quite capable of finding hundreds of laboratory 
customers who will claim satisfaction with services provided 
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70 Cross, "Report of the Laboratory Review Panel," p. 2. 

71 Chester Finn, Memo, Office of Educational Research and 
Improvement (October 27, 1987). 



by the laboratories. But I am saying that, given their 
present activities and configurations, and given the current 
fiscal constraints on the government, the laboratories 
represent a profligate use of OERI funds in relation to the 
benefit they generate. 

This is so for several reasons. I have already mentioned 
that laboratory impact is amorphous and difficult to assess, 
and that these institutions simply cannot provide services to 
more than a few districts in our immense education system. 
But congressional protection of the laboratories and, to be 
blunt, the insatiable appetite of the laboratories for federal 
funds, have shielded them from any real competition from other 
forms of dissemination and technical assistance. As a result, 

they have become entrenched institutions whose primary goal 

72 

seems to be self -perpetuation. 

Major Owens (D-NY) , Chairman of the Subcommittee on Select 
Education, shared some of Finn's reservations about the work of the 
labs and centers, but he questioned why OERI did not redirect the 
agenda and activities of the labs to be more productive and useful. 
When Finn failed to answer Owens satisfactorily, Milt Goldberg, the 
Director of PIP, stepped forward and the exchange between Goldberg 
and Owens illuminates the issue of OERI control ov^r the direction 

72 U.S. Congress, House, Subcommittee on Select Education, 

Oversight Hearings on the Offi ce of Educational Research and 

Improvement r PERIL 100th Congress, 2nd Session (House Document, 
Serial No. 100-77: Washington, DC: U.S. Government Printing Office, 
1988) , p. 171. 
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Mr. Goldberg ....Congressman Owens , there is a major 
difference in agenda setting between the labs and the centers , 
and that is that the lab agendas are set by their governing 
boards. We required, in the RFP's that were issued in 1985, 
that the laboratories establish governing boards that 
represent the major constituents in their regions, and that 
those governing boards set the work agendas for the 
laboratories. That differs quite considerably from the way 
the center agendas are set. 

Mr. Owens. Did you do that or did Congress do it? Did 
you have power that you gave away, or did the law require that 
you do it? 

Mr. Goldberg. No, the law did not require it. The 
Government did that. I mean, the administration did that. 

Mr. Owens. So in your regulations you gave away that 
power? 

73 

Mr. Goldberg. That's right. 

The growing skepticism and overt public hostility to the labs 
within OERI dissipated to a large degree when Cross replaced Finn 
as the Assistant Secretary in 1989. Having worked with the labs as 
chairman of the Laboratory Review Panel and having been employed 

73 U.S. Congress, House, Subcommittee on Select Education, 
Oversight Hearings on the office of Edu cational Research and 
Im provement , p. 244. 
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previously by the Subcommittee on Select Education, Cross accepted 
the fact that the labs would continue to be funded and sought to 
make modest improvements in their activities — particularly by 
expanding what the labs and centers did in terms of dissemination. 
No radical changes in either the direction or the management of the 
labs were envisioned. 74 

The National Academy of Sciences was funded to undertake a 
broad review of OERI and the legislative reauthorization of OERI 
was pending. Therefore, it was decided to make only minor changes 
in 1990 RFP since any larger alterations in the laboratory programs 
were expected to take into account the results from the National 
Academy study as well as from the directives in the reauthorization 
legislation. A number of small but significant changes in the 1990 
RFP were introduced such as reducing the disparities in regional 
funding, focusing the mission of the labs on "at-risk" students, 
serving small rural schools, emphasizing early childhood education, 
and allowing for greater flexibility in the delivery of services. 
Whereas the 1985 RFP stated that the amount of money spent on 
applied research should be approximately one-half of that spent on 
assistance, the 1990 RFP permitted labs greater freedom in 



74 Just before Cross became the Assistant Secretary, as 
chairman of the Laboratory Review Panel he submitted a final set of 
recommendations for the 1990 RFP for labs. Most of the 
recommendations had already been made in the 1987 report of that 
Panel. Many of the recommendations by the Panel were used in the 
writing of the 1990 RFPs. Christopher T. Cross, et al., "Report of 
the Laboratory Review Panel on the Pending Laboratory 
Recompetition," Office of Educational Research and Improvement 
(April 28, 1989) . 
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determining the appropriate mixture of expenditures on these 
tasks, 75 

Despite efforts to increase the amount of competition for the 
labs, only 11 eligible proposals were received for the 10 regions 
(one other application was ruled ineligible because it was received 
after the deadline) . Eight of the existing 9 labs were refunded 
(but not the one from the Southeastern Region which lost to the 
single competitor) and a new lab was created for the Pacific area. 
Interestingly, two of the 9 non-contested lab proposals were judged 
sufficiently weak and problematic by the panels that a third review 
was requested for those two lab proposals. 

II. Assessment of the Laboratories Today 

As with the analysis of the R&D centers, due to time 
constraints it was necessary to look at a sample of the labs. Five 
of the current 10 labs were examined in detail: 

1. Regional Laboratory for Educational Improvement of 
the Northeast and Islands (primary site at Andover, 
Massachusetts) [NE/I — also will be cited as the Northeast 
Lab] . 

2. Southwest Educational Development Laboratory (primary 
site at Austin, Texas) [SEDL — also will be cited as Southwest 

75 For an analysis of the changes in the 1990 RFP, see Charles 
Stalford, "Conduct of the 1990 Laboratory Competition," Office of 
Educational Research and Improvement (June 1991) . 
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Lab] . 

3. Mid-continent Regional Educational Laboratory 
(primary site at Aurora, Colorado) [McREL— also will be cited 
as Mid-continent Lab] . 

4. Northwest Regional Laboratory (primary site at 
Portland, Oregon) [NWREL— also will be cited as the Northwest 
Lab] . 

5. Far West Laboratory for Educational Research and 
Development (primary site at San Francisco, California) [FWL 
also will be cited as the Far West Lab] . 

The five labs to be investigated were suggested by the staff 

of the Educational Networks Division (END) of the Programs for the 

improvement of Practice (PIP) . The labs selected were intended to 

exemplify the current range and variety of research and development 

activities, but these institutions are not necessarily 

7 6 

representative of the other five labs which were not studied. 
Indeed, compared to the R&D centers the labs exhibit a much greater 
amount of diversity amongst themselves and therefore make it more 
difficult to generalize about the system as a whole from any 



76 Since it was important to look at labs which had Produced 
a considerable amount of research since 1985, it excluded the newly 
established Pacific Region Educational ^oratory <™> 
SouthEastern Region Vision for Education (SERVE) . 
investigated had a slightly larger budget than those that were not 
s?udied g The average budget for FY92 for the five labs studied was 
$3.7 million while that for the five other labs wa f s ^ 3 * *J the 
in larqe part due to the relatively small amount of funding for the 
Pacific Region Educational Laboratory (PREL) [$1.6 million per 



year] . 
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particular sample of them. On the other hand, by looking at 5 of 
the 10 labs rather than 5 of the current 23 centers, the proportion 
of the total number of institutions covered is much greater. 

The main focus of this analysis of the labs is on the quality 
of their research and development work. Since the labs engage in 
a wider variety of activities than the centers, it was sometimes 
difficult to decide what was appropriate for this investigation. 
One guiding principle was to look particularly et all of the task 
3 activities which encompass the applied research and development 
efforts of the labs. 77 Other research-related work, such as lab 
efforts to analyze a region's needs or to evaluate their own work, 
were also considered as well as any similar undertakings in their 
special initiatives (e.g. early childhood projects or rural 
education activities) . Moreover, any other materials or activities 
identified as research-related by the staff of the labs were 
analyzed. Because the emphasis was on the research-related 
activities and capabilities of the labs in the broadest sense, 
relevant lab work done with non-OERI funding and suggested by the 
staff of the labs was also investigated. Thus, some of the 
development work at the Mid-continent lab was analyzed even though 
it is no longer being financed by OERI. 

As with the analysis of the centers, all of the deliverables 
since 1985 from the five labs available from the staff at END were 
examined — including the published and unpublished research-related 

77 Task 3 included the applied research and development 
activities for the period FY91-FY95. Previously, these items had 
been included under task 4 for FY86-FY90 and were examined as well. 
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materials. Since past deliverables from two of the five labs were 

no longer maintained in END, it was necessary to reassemble those 

items from the labs. Due to a shortage of space at OERI when there 

is a change in institutional liaisons, sometimes some of the old 

78 

deliverables were discarded. The budgets of the labs were 

analyzed as well as their proposals for funding for 1985 and 1990. 
The written comments of the reviewers of these proposals as well as 
the communications from any internal or external evaluators of the 
labs was examined. The exchanges between the lab monitors and 
their contractors were also considered. All five labs were visited 
which provided an excellent opportunity to tour the facilities and 
meet with the staff. Each of the END institutional liaisons 
was interviewed several times and the frequent attendance and 
participation at the weekly PIP lab team meetings was invaluable. 
Written comments from several of the lab directors about the draft 
report were quite helpful. Finally, two meetings with all of the 
lab directors were held to discuss this report — including a meeting 
on April 7, 1993 after the labs had ample opportunity to review a 
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78 The products produced by the labs and centers are now 
being deposited in the U.S. Department of Education Library so that 
in the future anyone will be able to obtain them from that 
institution. 

79 The Northwest Lab was visited on August 31, 1992; the Far 
West Lab on September 1-2; the Northeast Lab on September 16; the 
Mid-continent Lab on September 28; and the Southwest Lab on 
September 29. While the agenda for these meetings varied somewhat 
from lab to lab, most of the time was devoted to meeting with the 
staff primarily responsible for the applied research, development, 
and evaluation activities . In addition , considerable time was 
spent with the directors of the labs and their executive committees 
and in some cases I had a discussion with all of the staff in a 
large meeting. 
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A. Distribution of Research-Related Activities of Labs 

As we have seen in the previous section, labs in the 1960s and 
1970s were expected to spend a large proportion of their funds on 
applied research or development activities. With the gradual 
elimination of large-scale curriculum development projects in the 
late 1970s and early 1980s and the relative de-emphasis of 
research-related activities in the 1985 and 1990 -lab competitions, 
the role of applied research and development in the labs has 
diminished. Indeed, some knowledgeable individuals initially 
wondered whether an analysis of the labs was not a wasted effort 
since they perceived that little research-related or development 
activities were being done today. 

Yet even a cursory glance at the budgets of the labs suggests 
that considerable funds still are being expended on just task 3 — 
the 'conduct of applied research and development. In FY92, the 10 
labs spent approximately $7.8 million (current dollars) of OERI 
funds on applied research and development while the R&D centers 
spent only an estimated $14 million on all of their OERI-funded 
research activities. Moreover, more than one-fifth (22.2 percent) 
of the lab budgets were allocated to task 3. Or, if one subtracts 
the indirect costs and fees of the labs, task 3 makes up 28.5 
percent of the remainder of the OERI funds to the labs. Only task 
2, providing assistance to the regions, receives more funding (32.9 
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percent of the total budget) . Thus, while the amount and 

percentage of OERI funds to the labs for applied research and 

development may have declined over time, it still remains a major 

and important expenditure for them, 80 

The percentage of the total budget money expended on task 3 

varies greatly among the labs for FY92 — ranging from 12.2 percent 

at the Pacific Lab (PREL) to 33.0 percent at the Northwest Lab 

(NWREL) (see figure 3). Overall, the five labs investigated spent 

an average of 22.4 percent of their total budget on task 3 

activities while the other five labs devoted an average of 22.0 

8 1 

percent of their total budget. 



Insert figure 3 



It is difficult to compare the direct and indirect costs of 
the labs and centers because they do not always categorize and 



For details on the FY92 budgets for the labs, see Charles 
Stalford, "Analysis of Laboratory Budgets," Memo, Office of 
Educational Research and Improvement (May 21, 1992). The FY92 
budget includes an additional $4 . 16 million Congressionally 
appropriated for the labs for the specific purpose of collaborating 
on a math and science initiative (under task four). If the $4.16 
million were eliminated from the overall FY92 lab budget, then task 
3 would be 25.2 percent of the total lab budget or 33.6 percent of 
the total lab budget not including indirect costs and fees. 

81 Some have argued that the labs have little control over how 
much money is spent on research and development because of the 
directives from OERI. While OERI's de-emphasis of research and 
development certainly has played an important role in how the labs 
have been able to respond to this area, the sizable variation among 
the labs on the percentage of funds spent on task 3 suggests that 
the labs maintain considerable discretion in the allocation of 
their funds. 

1 ^ * 
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subdivide their budgets in an identical manner. Nevertheless, it 

is interesting to observe that the indirect costs of the five 

centers studied were 27.6 percent while the indirect costs and fees 

for the 10 labs were 22.1 percent. Given the inexactitude of the 

figures, one should not place too much reliance on the specific 

differences, but it may be that the labs may spend slightly less on 

indirect costs and fees than the centers. The entire question of 

just what it costs OERI comparatively in indirect costs and fees to 

purchase services and products from the labs and centers warrants 

further investigation. 

Finally, we can look at the ability of the labs to obtain 

additional outside funding for their institutions. In recent 

years, OERI has been encouraging the labs to seek additional 

outside funding to supplement their budgets. This campaign appears 

to have succeeded as only one-half of the lab funding now comes 

from the regular OERI-lab budget. But there are great differences 

among the labs in their ability to garner outside funding. Two of 

the labs receive only about one- fourth of their monies from the 

OERI-lab budget, while three of them get about ninety percent or 

more from that source. Of the five labs investigated in this 

study, only 39.7 percent of their funds came from OERI-lab money 

compared to 73.6 percent of the funds for the other five 
8 2 

institutions . 

82 For information on the outside funding for the labs, see 
Stalford, "Analysis of Laboratory Budgets." Since that document 
did not have data on the outside funding for SEDL, that information 
was obtained directly from that lab (65 percent of their FY92 funds 
were from non-OERI-lab monies) . Much of the outside funds for the 
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There is some disagreement within OERI on the desirability of 
labs to seek more non-OERI funding. In the late 1960s and early 
1970s, labs sometimes were discouraged from seeking outside funding 
as this might divert their attention and energies from the needs of 
NIE. During the 1980s, however, the emphasis shifted toward 
encouraging outside funding — especially as it appeared to some that 
federal support of the labs might be terminated altogether. Given 
the changing attitudes toward outside funding by NIE/OERI over 
time, perhaps it is time for this agency to reconsider this issue 
once again. 

Since so much of the lab orientation and direction already is 
being set by the regional governing boards, the acquisition of non- 
OERI funding does not necessarily pose a problem at this time. If 
more of the funds and activities of the labs are focused on 
national priorities under the guidance of OERI, then the acceptance 
of large amounts on non-OERI funds potentially might pose a problem 
for the allocation of lab staff and resources. On the other hand, 
additional future funding might allow the labs to make more 
efficient and effective use of the OERI monies through the cost- 
sharing of projects and the use of federal education to build upon 
work initially sponsored by others. Therefore, it is unlikely that 
there is any simple formula or rule that should be followed in 
regard to an ideal overall portfolio of lab funding. Perhaps the 
important point to bear in mind is that OERI should always strive 



labs are from other sources in the U.S. Department of Education 
(including some funds administered by other programs within OERI)* 
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to make sure that the federal interests in lab activities are 
protected even as these institutions simultaneously respond to the 
needs of their other funders. 

B. Research Topics of Labs 

We have already seen in the previous section that a 

considerable portion of lab funds were spent on task 3 — applied 

research and development. Another perspective on overall lab 

expenditures is provided by looking at the distribution of their 

activities as recorded by the labs on OERI's Project Input Forms 

8 3 

(or PMIS forms) for FY92. According to these forms, activities 
were subdivided into seven categories: (1) policy studies, (2) 
evaluation, (3) basic research, (4) applied research, (5) 
development, (6) statistics, and (7) dissemination. Unfortunately, 
compared to similar data provided by the centers, the labs overall 
do not appear to have been as careful and accurate in returning 
this information to OERI. Nevertheless, it is worth glancing at 
the PMIS returns to obtain a rough idea of the self-reported 
activities of the labs. 

Although the quality of the data from the PMIS forms leaves 
much to be desired in terms of categorization and accuracy, the 
overall contours of the five labs 1 description of their own 



The Pr o j ect Input Forms are part of the OERI Pro j ect 
Management-Information System (PMIS) . Therefore, sometimes these 
forms will be referred to simply as PMIS forms. 
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8 4 

activities are instructive. According to the PMIS forms for 

the five labs, these institutions devote considerable funds and 

energy to dissemination. Approximately 32.6 percent of their total 

OERI funds is spent on dissemination. Unfortunately, PMIS does not 

provide more detailed information about the nature of that 

dissemination, but in the case of the labs it appears to consist 

mainly of directly providing information to clients as well as of 

85 

offering technical assistance to them. 

Using the information from PMIS on the distribution of 
activities (but excluding dissemination which has been discussed 
above) , we can see that labs devote very little effort to basic 
research (3.1 percent), evaluation (10.2 percent), or statistics 
(1.2 percent). While according to the PMIS forms policy studies 
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The categories of the PMIS forms might have been improved 
if there had been added an administrative/managerial subgroup and 
if some of the current categories such as development had been 
defined more precisely or subdivided further to make some useful 
distinctions. 

For the purposes of this analysis, it seemed appropriate to 
remove the indirect costs and the fees from each of the projects in 
order to make the results more similar across the labs and more 
comparable to the analysis of the centers. Moreover, since most of 
Task 1 was related to managerial and administrative activities, it 
was also removed from the analysis whenever possible. It should 
also be noted that the FY92 PMIS forms contained information from 
Task 6 (Early Childhood Education Linkages) even though this was 
actually funded by DHHS through OERI but the FY92 PMIS information 
did not include any data from the $416,000 supplement for each of 
the labs for collaboration on math and science. 

85 While a considerable portion of lab dissemination 
activities are providing technical assistance to clients, not all 
service to clients (task 2) are categorized as dissemination by the 
labs. For example, the Mid-continent Lab (McREL was one of the few 
labs to fill out its PMIS forms by tasks) subdivides task two as 10 
percent policy studies, 60 percent applied research, and 30 percent 
dissemination. Overall, according to PMIS, 85 percent of McREL 1 s 
total dissemination budget was expended under task 2. 
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also receive relatively little attention (12.7 percent), that 
figure may underestimate this activity because some of the policy 
analyses appear to have been placed under the applied research 
category as well (see figure 4) . 

Insert figure 4 

The two major lab activities according to the PMIS forms are 

applied research (27.7 percent) and development (45.1 percent). 

Together they account for almost three-fourths of lab research and 

development-related activities. Compared to similar data for the 

centers, the labs proportionately were more likely to spend funds 

on development while the centers focused more on applied research, 

evaluation, basic research, and statistics (see figure 5). They 

spent about equal proportions on policy studies (though the labs 

probably would have a higher proportion here if the policy studies 

8 & 

categorized under applied research were also included) . 

Insert figure 5 

One of the problems with the use of the PMIS categories is 



8 6 

The comparisons of the distribution of expenditures between 
the centers and labs must be seen as very crude and inexact — even 
though both institutions use the same PMIS forms. For example, the 
centers exclude most of their management expenses (beyond the 
indirect costs) from their PMIS breakdowns while the labs probably 
still include some of them. As a result, while comparisons of 
relative types of expenditures between the labs and centers are 
interesting and useful, they should be seen as only rough 
approximations to their actual expenditures. 
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that they are often employed for rather different purposes by the 
respondents. For example, vhile sometimes applied research at the 
labs refers to efforts to conduct original research on some 
specific problem, at other times it refers to activities to 
synthesize existing research or to write policy papers based upon 
existing research. Indeed, much of the applied research done in 
the labs is really the synthesis or use of existing applied or 
basic research rather than the support of original research. 

In the previous essay we observed how many of the R&D centers 
relied heavily upon case studies for their applied research 
investigations. We also discussed the strengths and weaknesses of 
the use of case studies. While individual case studies can be an 
effective means of initially exploring some problem or of later 
looking at something in more depth, one can only draw limited 
conclusions from their findings since they are not representative 
of the population as a whole. Moreover, unless individual case 
studies are placed in some kind of contextual or comparative 

framework, it is usually difficult to draw any meaningful research 

87 

or policy inferences from them. 

The labs frequently relied on case studies as a means of 
conducting their applied research. Often this meant looking at a 
particular institution or process that was relatively isolated from 



For discussions of the use of case studies, see Charles C. 
Ragin and Howard S. Becker, eds. , What is a Case? Exploring the 
Foundations of Social Inquiry (Cambridge: Cambridge University 
Press, 1992); Joe R. Feagin, Anthony M. Orum, and Gideon Sjoberg, 
eds., A Case for the Case Study (Chapel Hill, NC: University of 
North Carolina Press, 1991). 
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their other research or development endeavors and thereby limited 
in its overall usefulness and generalizability • For example, the 
Northwest Lab analyzed the grading practices of 15 teachers within 
a particular high school and properly acknowledged the limitations 
of generalizing the findings from this single case study, 88 

Occasionally efforts were made to embed case studies as part 
of a larger research effort. The Northeast Lab, for example, was 
one of the primary sponsors in the 1980s of studies which sought to 
understand and to improve urban high schools. The Northeast Lab 
commissioned a telephone survey of 178 urban high schools which had 
introduced some significant educational innovation. It also 
sponsored five indepth, qualitative case studies of innovative 
urban high schools or junior high schools (in Boston, Cleveland, 
Los Angeles, New Jersey, and New York City) . Based upon the 
results from the survey and the five case studies, explanations 
were offered on why and how innovations introduced into the public 
secondary schools in urban communities succeeded or failed in 
helping to improve student learning outcomes. 

Similarly, the Southwest Lab (SEDL) sought to improve rural 
education by working with a small , rural school Serving 



Richard J. Stiggins, Philip Griswold, and David Frisbie, 
"Inside High School Grading Practices, M Northwest Regional 
Educational Laboratory (September 1990).. 

89 

For a summary and detailed discussion of this project, see 
Karen Seashore Louis and Matthew B. Miles, Improving the Urban High 
School: What Works and Why (New York: Teachers College Press, 
1990) . Although the Northeast Lab provided the primary funding for 
this project, the North Central Lab and the Far West Lab also 
contributed some assistance. 
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disadvantaged students in five different states (Arkansas, 
Louisiana, New Mexico, Oklahoma and Texas). While not all of these 
demonstration sites used identical achievement tests or employed 
the same set of contextual indicators, an effort was made to 
compare the five case studies whenever possible. 

Much of the work categorized as applied research by the labs 
consists of syntheses of existing research or of writing policy 
papers which incorporate relevant research findings . 91 The 
topics for these papers range widely from an understanding of 

language development and education to concern about school-wide and 

92 

classroom discipline. However, there is sometimes considerable 
duplication among the laboratories in their research syntheses and 
policy papers. For example, many of the labs, often as part of 
their task 6 activities, have produced research summaries or policy 
papers about early childhood education. While some of them do have 
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For a description and analysis of this project, see Deborah 
V. Jolly, Shirley M. Hord, and Marianne Vaughn, "Developing 
Indicators of Educational Success: The Road to Improvement in Five 
Schools," Paper presented at the American Educational Research 
Association Annual Meeting, Boston, April 1990. 

91 

Using task 3 funds for supporting research syntheses was 
explicitly sanctioned and encouraged in the Request for Proposal 
(RFP) for labs in 1990. One of the five illustrations of 
permissible activities under task 3 was "synthesizing R&D, or 
otherwise contributing to knowledge about the improvement of 
schooling, particularly for at-risk populations." Programs for 
Improvement of Practice , OERI , "Regional Educational Laboratory 
Request for Proposal" (Washington, DC: U.S. Department of 
Education, 1990) , p. 24. 

92 

Nancy Faires Conklin, Carole Hunt, and Laura Walkush, 
"Language Development: A Base for Educational Policy Planning," 
Northwest Regional Laboratory (July 1990) ; Kathleen Cotton, 
"Schoolwide and Classroom Discipline, " Northwest Regional 
Educational Laboratory (October 1990) . 
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a particular regional slant or orientation, many of them easily 

might have been equally applicable and useful in another part of 
93 

the country. 

OERI and the labs should re-examine how research syntheses and 
policy analyses are produced and distributed. Sometimes the labs 
commission their own research or policy papers* At other times 

they use the one-page summaries developed by the Council for 

♦ 94 
Educational Development and Research (CEDaR) . Occasionally the 

labs also use directly materials developed or funded by OERI. 

There appears to be considerable confusion, duplication, and 

waste in how research syntheses and policy papers are developed at 

OERI and the labs and centers. OERI already sponsors several 

different activities in this area besides funding the extensive 

work of the labs and centers. The Office of Research (OR) has 

initiated an Education Research Guide series which provides a four- 
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Janet Jewett, "Effective Strategies for School-based Early 
Childhood Centers," Northwest Regional Educational Laboratory 
(December 1991) ; J. Ronald Lally and Peter Mangione, "Early 
Intervention Research: Building on Lessons from the 60* s and 70 f s 
For Programs in the 90's, H Paper presented at Conference on New 
Directions in Child and Family Research: Shaping Head Start in the 
Nineties, Sausalito, CA, June 1991; Kenneth H. Hansen, "Early 
Childhood Education: Policy Issues," Northwest Regional Educational 
Laboratory (May 1988); McREL, "Early Childhood Education," Policy 
Notes, 4, No. 1 (Spring 1990). 

94 

The labs, using OERI funds, subcontract with CEDaR for the 
production of these one-page summaries. The labs then disseminate 
the CEDaR summaries to their regional clients (sometimes adding 
some of their own research/policy summaries) , After receiving 
research and policy reports from the labs , centers , and other 
scholars or organizations, CEDaR evaluates them for quality and 
relevancy in deciding which ones to use for their one-page 
summaries. While many of the OERI-funded R&D centers send their 
materials to CEDaR, some have refused to participate — perhaps not 
realizing that this is an OERI-funded activity. 
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page summary of topics such as cooperative learning, performance 
assessment, and reading recovery. The National Center for 
Education Statistics (NCES) provides publications such as NAEPfacts 
and Education Research Report , In addition, NCES has also begun a 
two-page Issue Brief series. Finally, the Educational Resources 
Information Center (ERIC) sponsors 16 ERIC Clearinghouses which 
develop voluminous short research and policy syntheses, 
bibliographies, digests, and books for parents, researchers, and 
practitioners. Therefore, before an individual lab or center 
commissions a research synthesis or policy paper, it may be prudent 
to check to see whether a roughly comparable product already 
exists. Moreover, OERI should co-ordinate its own publication 
activities and perhaps an arrangement sometimes could be developed 
whereby topics that are of special interest to the labs or centers 

might become one of the priorities for OERI's other ongoing 

. . . 95 
activities. 

After consulting the existing stock of research syntheses or 
policy analyses, if a need for more work still exists (or if the 
existing products are deemed inappropriate or inadequate) a lab or 
center might want to commission additional papers. In some 
situations, however, the lab or center still might find it more 
economical and effective to persuade some other OERI-funded group 
to undertake that work because their staff may be more experienced 

95 

When Christopher Cross was the Assistant Secretary, plans 
were being made to co-ordinate all of the publication activities 
undertaken or funded by OERI. Unfortunately, those plans never 
materialized and the entire co-odinating effort was subsequently 
abandoned . 
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and knowledgeable in this area. In any case, the entire issue of 
producing and disseminating research syntheses and policy analyses 
needs to be thoroughly reconsidered. 

While more than one out of every four lab dollars is spent on 
development, there is little agreement on what that term means. 
Sometimes the category "development" is used to cover activities 
such as task 1 — the establishment of effective governance, 
management and planning systems. 96 At other times the term 
"development" is used to describe the creation of classroom 
materials and the training of teachers and professionals. Only 
rarely was the word "development" associated with the more classic 
and traditional definition—the repeated and systematic testing and 
improvement of some education product or curriculum. 

Given the common usage of the term "development", it is not 
surprising that the labs employed the concept so broadly. Nor is 
it necessarily bad as long as everyone understands exactly what is 
meant when someone uses that term. Unfortunately, there is so much 
confusion and uncertainty surrounding the word "development" today 
that by itself the word is not an effective means of communicating 
what is being done. 

When the labs were created in the mid-1960s, the Gardner Task 
Force and others assumed that these institutions would play a major 
role in the development of education curricula and materials. The 
Gardner Task Force envisioned development in the more traditional 

For example, the Mid-continent Lab (McREL) subdivided all 
of its activities for the PMIS according to the six tasks. For 
task one they designated it as 100 percent development. 
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sense — the use of iterative and systematic testing to improve 

curricula or other educational products using large-scale field 

studies. Indeed, the Gardner Task Force insisted that all of the 

labs have attached to them experimental schools as one element of 

97 

systematically assessing and improving their products. 

Many of the labs in the 1960s and early 1970s did employ this 
concept of development. For example, the Southwest Lab (SEDL) 
developed a short handbook which described the development process 
for educational products. They broadly defined an educational 
product to "comprise instructional materials, hardware, or 

software; it can comprise a technique or a process; or it can 

98 

comprise any combination of the above." 

SEDL then described six stages of the developmental cycle: (1) 
context analysis, (2) conceptual design, (3) product design, (4) 
pilot test, (5) field test, and (6) marketing and diffusion. 
Educational products did not have to proceed linearly through the 
stages (products adapted from elsewhere usually were introduced in 
a stage other than the first one). SEDL's product development 
model was not inflexible and could vary from one product to the 
next. But throughout the entire process, the emphasis was on 
continuing evaluation and testing of the educational product. A 
product was pilot tested usually "under controlled conditions in 



97 Gardner, "Report of the President's Task Force on 
Education." 

98 Robert Randall et al., "A Developmental Process Adopted by 
the Southwest Educational Development Laboratory, July 14, 1970," 
Southwest Educational Development Laboratory (1970), p. 4. 
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selected schools which are in proximity to the Laboratory 
and... conducted by the originators of the test products." 99 Then 
the product was subjected to large-scale field testing "to 
determine the ultimate utility and viability of the system under 
test, and to facilitate marketing and diffusion of the system by 
measuring its effectiveness, cost, endurance, and potential and by 
ascertaining the effects upon the system of the many variables 
existing in a natural environment." 100 

SEDL was not alone among the early labs in its interest in the 
systematic development of educational products. Most of the 
developmental energies in the labs in the 1960s and 1970s were 
devoted to improving the curriculum. As was discussed previously, 
NIE opposed large-scale, curriculum development in the late-1970s 
and OERI explicitly prohibited it in the Reguests for Proposals 
(RFPs) for labs in 1985 and 1990. Nevertheless, a few of the labs 
managed to continue some curriculum development— though at least 
one of the labs felt it was prudent to create a separate, 
subsidiary corporation in order not to violate OERI's prohibition 
against large-scale curriculum development activities. Thus, the 
Mid-continent Lab (McREL) used OERI funds to support the research 
phases of a major curriculum project and then completed the 
developmental phase of this activity using their other 



99 
100 



Randall et al., "A Developmental Process," p. 25, 
Ran' 11 et al., "A Development Process", p. 29. 
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corporation's monies. 

If a few large-scale curriculum development projects have been 
continued at the labs using non-OERI funds, most of the labs have 
turned to smaller projects and used less systematic ways of 
assessing and improving their products. The Northwest Lab, for 
example, has been developing its Onward to Excellence Program for 
over a decade, but has never undertaken an extensive and rigorous 
large-scale field test of the project. Instead, the Onward to 
Excellence Program utilizes the experiences of teachers and other 
professionals who have used the program to make any adjustments and 

. 102 
improvements . 

According to the guidelines in the RFPs for the 1985 and 1990 
lab competitions, the expectation was that labs would undertake 
only short-term applied research or development projects (ones that 
could be completed within the 5-year contract period) . There was 
no emphasis placed on developing large-scale ongoing cumulative 
projects—in fact such efforts implicitly seem to have been 
discouraged. Therefore, it is not surprising that many of the 



101 Robert J. Marzono, et al., Dimension s of Thinking; A 
Framework for Curriculum and Instruction ((Alexandria, VA: 
Association for Supervision and Curriculum Development, 1988); 
Robert J. Marzano, A Differe n t Kind of Classroom; Teaching with 
Dimensions of Learning (Alexandria, VA: Association for Supervision 
and Curriculum Development, 1992); Robert J. Marzano, et al., 
implementing Dimensi o ns of Learning (Alexandria, VA: Association 
for Supervision and Curriculum Development, 1992) . 

102 For a discussion of the Onward to Excellence Program, see 
Robert E. Blum, Kim O. Yap, and Jocelyn A. Butler, "Onward to 
Excellence Impact Study," Paper presented at the American 
Educational Research Association Annual Meeting, San Francisco, 
April 1992. 
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applied research or development projects were small and often 

unrelated investigations. 

The regional focus of the applied research also limited the 

ability or willingness of some of the labs to use OERI funds to 

investigate questions at sites outside their immediate geographic 

area. For example, the Northeast Lab obtained a sizable grant from 

Apple Computer Corporation to study the impact of computers in the 

classroom. When the funding for this interesting and important 

project abruptly and unexpectedly ended, the Northeast Lab was 

unwilling to use OERI-funds to complete this study because the 

actual classroom sites for the project were outside their 

region. Others such as the Far West Lab, however, do not 

feel quite as territorially bound by the confines of their region 

104 

and are more willing to operate on a national basis. 

Although the present labs are about one-fourth smaller in 
constant dollars than in the early 1970s, the reduction in funding 
does not really explain the small size of most of their applied 
research and development projects. Once we eliminate the indirect 



For a description and discussion of the project on 
computers in the classroom, see Diane Kell, Glen Harvey, and Nancy 
Gadzuk Drexler, "Educational Technology and the Restructuring 
Movement: Lessons from Research on Computers in Classrooms," Paper 
presented at the American Educational Research Association Annual 
Meeting, 1990. 

104 For example, the Far West Lab undertook a multi-site case 
study evaluation of 24 Chapter I schools throughout the country 
with funding from OERI (but not from OERI's regional lab budget). 
Brian Rowan et al., "The Design and Implemantation of Chapter 1 
Instructional Services: A Study of 24 Schools," Far West Lab 
(November 1986) ; Brian Rowan and Larry F. Guthries, "The Quality of 
Chapter 1 Instruction: Results from a Study of 24 Schools," Far 
West Labs (March 1988). 
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costs and fees, the administrative expenses , and dissemination 
expenditures, the average amount of money directly available for 
applied research and development (task 3) for each of the labs in 
FY92 is substantially more than is available to most of the 
individual R&D centers. But overall the labs choose to spend less 
of their task 3 funds on original applied research and development 
than did most of the R&D centers. Moreover, even given the small 
amount of money that the labs devote to original applied research 
and development, they generally also prefer to spread the limited 
funds among several small-scale and short-term projects (much like 
many of the R&D centers) . 

C. Quality of the Applied Research and Development 

Almost any discussion of the quality of applied research and 
development produced by the labs evokes strong feelings— often by 
those who are not personally very familiar with much of the work at 
those institutions. Many academics and policy makers are 
particularly scornful of the quality of the research and 
development at the labs and see them as second-rate institutions 
compared to the R&D centers or the funding of individual 
researchers. The strong antipathy towards the quality of the 
research and development produced by the labs even extends to some 
OERI staff and leaders (though not among most of the END staff who 
actually monitor the work of the labs). Moreover, there is a 
tendency among many critics of the labs simply to issue a blanket 

173 
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condemnation of the research and development of the labs — in part 
because of the widespread perception that the congressional 
earmarking of the funds for these institutions has come directly at 
the expense of field-initiated research. 

Part of the skepticism of the quality of the work of the labs 
may be a self-fulfilling prophecy because of OERI directives in the 
1985 and 1990 lab competitions discouraged long-term applied 
research or development projects and emphasized providing immediate 
technical assistance to clients. Therefore , it should not be 
entirely surprising that many policy makers and academics now have 
difficulty in finding original lab research or development projects 
worth praising. 

Nevertheless, some of the labs since 1985 have produced some 

first-rate applied research and development. Particularly 

impressive has been the work at the Far West Lab. Building upon 

their long tradition of excellent research, the Far West Lab 

continues to produce some of the more innovative and interesting 

applied lab research and development today — often funded with non- 
105 

OERI monies. 

The Far West Lab specializes in fairly sophisticated and 
thoughtful policy analyses. Using funds from both OERI and the 

105 Under pressure from NIE (and its successor, OERI) in the 
early 1980s, the Far West Lab moved away from its emphasis on basic 
research to applied and developmental work. There was a 
significant shift in staffing away from the more academic and pure 
research to more applied and developmental work. While the current 
staff are less oriented toward academic and basic research than 
their predecessors, many of them appear to be relatively well- 
trained and sophisticated in their methodological and statistical 
skills. 
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Utah State Office of Education, they evaluated the Utah Career 
Ladder System for elementary and secondary public school teachers* 
The state's 40 superintendents and school board presidents were 
interviewed by telephone* Surveys were mailed to all of the 
principals in the state as well as to a random sample of 1500 
teachers. 106 They also analyzed the teacher salary distribution in 
10 districts and investigated the implementation of the Career 
Ladder System in 12 districts. Unfortunately, they did not collect 
any data on changes in student outcomes; but on the basis of their 
interviews with principals and teachers, they concluded that "the 
policy is powerfully and positively changing both the teaching 
profession and the ways schools are organized to teach students. 
Utah's Career Ladder System is a model that deserves national 

4-4- 4.- ,.107 

attention." 

Similarly, the Far West Lab analyzed the impact of the 
introduction of a computerized, self -paced math program (the so- 
called Rubin program) in five elementary schools and one middle 
schools. Using information from a five-year longitudinal study of 
the standardized math achievement scores of those students, they 
analyzed the scores of those who had been in the program for 

106 The response rate for the mailed questionnaires to the 
principals and teachers was 68 percent and 63 percent respectively. 
Unfortunately, they did not try to ascertain what biases might have 
been introduced by this response rate. Mary Amsler, Douglas 
Mitchell, Linda Nelson, and Thomas Timar, "An Evaluation of the 
Utah Career Ladder System: Summary and Analysis of Policy 
Implications," Far West Laboratory for Educational Research and 
Development (January 1988), p. 3. 

107 Amsler, Mitchell, Nelson, and Timar, "An Evaluation of 
the Utah Career Ladder System," p. 2. 
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varying lengths of time. The results of this study provided only 

a lukewarm endorsement of the Rubin program, "[P]rimary students 

in the Rubin program did less well than their regular program 

counterparts during the first two years of implementation 

but. .. intermediate grade students did better. Growth rates were 

higher for students in the Rubin program for four years than 

108 

students in the program for two years," 

Final ly, the Far West Lab sponsored one of the more 

sophisticated and indepth policy reviews of the relationship 

between class size and student learning. Based upon an extensive 

and careful review of the existing secondary literature, they 

concluded that "reducing class size has a substantial and 

cumulative effect on student learning/ 1 but that "the costs of 

109 

class size reductions are enormous." Using data from a large 

number of studies of class size, they found a curvilinear 

relationship between class size and achievement and then tried to 

110 

develop a theoretical model to explain the pattern. 

None of the other four labs in this investigation produced the 
same overall quantity or quality of applied research and 
development as at the Far West Lab. Except for the Northwest Lab, 



Robert B. Burns, "Longitudinal Analyses of Student 
Achievement in an Innovative Mathematics Program, 11 F^r West 
Laboratory for Educational Research and Development (July 1988). 

109 Douglas Mitchell, Crist i Carson, and Gary Badarak, "How 
Changing Class Size Affects Classrooms and Students," California 
Educational Research Cooperative (May 1989), pp. 67-68. 

110 Mitchell, Carson, and Badarak, "How Changing Class Size 
Affects Classrooms and Students," p. 68. 
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the Far West Lab devoted more of its OERI funds on task 3 (and 
Northwest spent a larger proportion of its task 3 monies on 
dissemination rather than on applied research or development) . 
Moreover, with some notable exceptions, most of the applied 
research produced at the other four labs was not as conceptually or 
methodologically rigorous or sophisticated as that prepared by the 
Far West Lab. 

Yet there were several applied research or development 
projects at the other four labs which were praiseworthy. For 
example, the Northeast Regional Lab, using funds from the Apple 
Corporation, produced a useful assessment of the computer-based 
Apple Learning Series: Early Language (ALS-EL) in elementary 
classrooms. 111 The Southwest Regional Lab produced an important 
six-year longitudinal study of language and reading achievement in 
bilingual classrooms. The Mid-continent Lab continues to do 

valuable development work in vocabulary instruction and curriculum 
frameworks. 113 And the Northwest Lab is doing a longitudinal 
analysis of Washington State f s Early Childhood and Assistance 
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111 Glen Harvey, Diane Kell, and Nancy Gadzuk Drexler, 
"Research on Computers and Literacy Development in Primary 
Classrooms," Final Report, Regional Laboratory for Educational 
Improvement of the Northeast and Islands (July 1990) . 

112 Betty J. Mace-Matluck, Wesley A. Hoover, and Robert C. 
Calfee, "Teaching Reading to Bilingual Children: A Longitudinal 
Study of Teaching and Learning in the Early Grades," National 
Association of Bilingual Educators Journal, 13, 3 (1989), 187-216. 

113 Robert J. Marzano and Jana S. Marzano, A Cluster Approach 
to Elementary Vocabulary Instruction (Newark, DE: International 
Reading Association, 1988) ; Marzano, et al. , dimensions — of 
Thinking ? Marzano, A Different Kind of Classroom ? Marzano, et al., 
Implementing Dimensions of Learnin g. 



JL * g 



166 

114 

Program. 

But there are also serious conceptual and statistical 
limitations of some of the work supported by the labs. Some of the 
applied research appears to be very preliminary and highly 
descriptive."**"" Major projects sometimes iacK any Ciear 
research designs. The Northeast Lab launched an intensive 
Collaborative Action Research effort in three sites, but did not 
have an adequate research design for that large-scale and long-term 
undertaking . 

The statistical analyses of some of the studies are flawed and 
misleading. Thomas Owen and Carolyn Cohen investigated the 
attitudes and opinions of entry-level workers ^in four states: 
Hawaii, Idaho, Oregon, and Washington. Each of the states appears 
to have used a different strategy for sampling companies and 
workers and the overall completion rate was extremely low (20 
percent for the workers and 29 percent for the companies) . 
Moreover, the completion rates for the four states varied 



14 Child, Family, and Community Program, ECEAP 1991 
Longitudinal Study and Annual Report : Washington State t s Early 
Childhood Education and Assistance Program . Northwest Regional 
Educational Laboratory (1991). 

115 R. G. Schwab, Sylvia Hart-Landsberg, and Karen Reed 
Wikelund, "Implementation Strategies for Innovative Literacy 
Practices: A Regional Depiction, 11 Northwest Regional Educational 
Laboratory (April 1991) . 

116 For details of the Collaborative Action Research effort, 
see David P. Crandall, "Annual Report: 1 December 1990-30 November 
1991," Regional Laboratory for Educational Improvement of the 
Northeast and Islands (December 1991) . As of June 1993 the 
Northeast Lab still had not submitted any detailed research plans 
for their Collaborative Action Research project. 
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considerably. Yet the investigators simply combined the results 

from the four states without adjusting the data for the unequal 

return rates. Despite all of these shortcomings, they claim that 

the "data might also be used as a benchmark from which to measure 

117 

change in future years." 

Similarly, McREL sent questionnaires to local public school 
superintendents in the seven states it serves. The overall 
response rate was only 40 percent and it varied among the states 
(ranging from 27 percent for Nebraska superintendents to 53 percent 
for Wyoming superintendents) . Despite the considerable variation 
in response rates, the analysts simply grouped all of the returns 
together to get an overall regional prof ile— even though this meant 
that a state like Nebraska which had 27 percent of the region's 
superintendents only contributed 19 percent of the returned 
questionnaires. If there were no variation in answers to the 
questions by states, then this indiscriminate totalling of the 
responses would not affect the results. But often there were clear 
and sizable differences in the answers of the superintendents to 
the questions. As a result, the detailed tables in the McREL 
Policy Notes which show whether or not the answers from the 
superintendents from a paiticular state are statistically 
significantly higher or lower than the regional mean may be 
inaccurate if we recalculated the regional means based upon the 
questionnaires stratified by the rate of return. In any case, 

117 Thomas Owens and Carolyn Cohen, "Northwest Entry-Level 
Worker Study," Northwest Regional Educational Laboratory (September 
3G, 1991) . 
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given the low and unequal response rate to the questionnaires, the 

^ 118 
analysts should have addressed these problems more adequately. 

Although the evaluation of programs (including their own) is 

an important part of the work of the labs, often these assessments 

were not well-designed. The Northwest Lab study of the impact of 

its interesting and important Onward to Excellence (OTE) Program 

could have been improved. The analysts did not explain why certain 

OTE programs were chosen in the impact study while others were not 

included. 119 Moreover, by looking indepth at only 10 of the 

more successful OTE programs, they could not ascertain why some of 

them succeeded and some of them failed. One might have also looked 

at some schools which did not join the OTE program in order to be 

able to t»ee if any of the improvements in the OTE schools were 



118 McREL, "Status of Education in the McREL Region," Policy 
Notes, 4, No. 3 (Winter 1990). 

119 There were 625 schools in 1990 which participated in the 
Onward to Excellence (OTE) Program; 292 had been involved with OTE 
for at least two years and therefore were eligible for 
participation in this impact study. Of these, 163 were in the 
School Improvement Network Directory which became the initial 
database for this study. Naturally, one wonders what kind of 
biases might have been introduced by looking only at those schools 
which joined the School Improvement Network. Perhaps one could 
sample some of those which did not join to see if they were any 
different than those that did join. Or perhaps it might make even 
more sense to develop an ongoing longitudinal analysis of all 
participants in the Onward to Excellence Program so that it could 
be used for both analysis and monitoring (a standardized 
questionnaire might be developed so that each program could 
routinely complete it ea*jh year) . For details of the impact study 
of the Onward to Excellence Program, see Robert E. Blum, Kim O. 
Yap, and Jocelyn A. Butler, "Onward to Excellence Impact Study," 
Paper presented at the American Educational Research Association 
Annual Meeting, San Franoisco, April 1992; Robert E. Blum, Kim 0. 
Yap, and Jocelyn A. Butler, "Onward to Excellence Impact Study," 
Northwest Regional Educational Laboratory (February 1993). 
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partly the result of more general changes affecting all schools 

during those years. Finally, rather than only using a definition 

of "success" based on whether the program achieved one of its 

goals, the definition might have been expanded and made more 

standardized. Otherwise, OTE programs which set low goals might be 

defined as equally or even more "successful" than those programs 

120 

which had set higher goals. 

A typical shortcoming of many of the lab demonstration and 
evaluation projects was the lack of standardized measures that made 
comparisons meaningful or the failure to include any appropriate 
controls in their research designs. For example, the Southwest Lab 
investigated small and economical ly-disadvantaged rural schools in 
five different states ever several years. The five demonstration 
schools, all of which had indicated an interest in improving their 
student achievement scores, were visited each month by a SEDL staff 
member responsible for facilitating change in that institution. 
Changes in education indicators over time were used to assess the 
improvements. Unfortunately, since the five sites did not use any 
overall standardized measures of educational success, it was 
impossible to make any systematic comparisons among the five 
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120 interestingly, when I discussed the conceptual and 
methodological shortcomings of this study with one of its authors, 
he readily agreed and explained that they had been constrained by 
the lack of time and funds available for this particular analysis. 
However, when I discussed the same issue with those at the 
Northwest Lab responsible for evaluation, they defended the study 
as methodologically sound and adequate. When the Northwest Lab 
submitted their final study, they did not seem to acknowledge or 
discuss the methodological shortcomings of their study that had 
been raised in writing with them by OERI and myself. Blum et al., 
"Onward to Excellence Impact Study" (1993). 
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schools. Since it appears that the schools which chose to 
participate were not necessarily typical of others in that region, 
it is difficult to know how much we should generalize from these 
findings* Moreover, since there were no control schools in the 
study we cannot establish just what factors were instrumental in 
making any of the improvements in student achievements. Thus, the 
SEDL project basically remains a limited but interesting 
demonstration effort at five diverse sites; unfortunately, their 
work does not provide the type of information that would be useful 

in ascertaining more generally or conclusively what are the key 

_ . ^. 121 
factors in improving rural education. 

Much of the research-related work of the labs consists of 

summarizing and synthesizing existing materials for policy papers. 

Some of these policy briefs are quite well done and very useful. 

The Northeast Lab, for example, has put together a nice series of 

briefing papers on teacher quality, pension portability, and 

teacher incentives. 122 The Mid-continent Lab has also produced 

some useful policy papers on subjects such as early childhood 



Deborah V. Jolly, Shirley M. Hord, and Marianne Vaughan, 
"Developing Indicators of Educational Success: The Road to 
Improvement in Five Schools," Paper presented at the American 
Educational Research Association Annual Meeting, Boston, April 
1990. 

122 Anne E. Newton, "Teacher Quality: An Issue Brief," 
Regional Laboratory for Educational Improvement of the Northeast 
and Islands (May 1987) ; David Title, "Pension Portability in the 
Northeastern States," Regional Laboratory for Educational 
Improvement of the Northeast and Islands (1990); David Title, "The 
Critical Role of Teacher Incentives in the Northeast States," 
Regional Laboratory for Educational Improvement of the Northeast 
and Islands (March 1989). 



171 

123 

education. 

But some of the briefing or policy papers are simply a loose 

catalogue of research without adequate attention either to the 

quality of the work being cited or the diversity of materials in 

the field. Kenneth Hansen 1 s policy paper on early childhood 

education, for instance, does not cite or use the existing research 

124 • 

literature on this subject very effectively. Similarly, 

Kathleen Cottons glowing and uncritical discussion of the benefits 

of early childhood education ignores the studies which question the 

12 5 

efficacy of programs such as Head Start. Janet Jewett's 

review of research states that "[s]tudies recommend limiting 

elementary class sizes to 15 in public schools and stress small 

126 

group size for preschool children."^ In fact, most of the 



studies of class size in elementary schools present a much more 

complex picture — including a publication from the Office of 

127 

Educational Research and Improvement (OERI) three years earlier. 
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123 Mid-continent Regional Educational Laboratory, "Early 
Childhood Education," Policy Notes , 4, 1, 1-11. 

124 Kenneth H. Hansen, "Early Childhood Education: Policy 
Issues," Northwest Regional Educational Laboratory (May 1988). 

125 Kathleen Cotton, "Educating Urban Minority Youth: Research 
on Effective Practices," Northwest Regional Educational Laboratory 
(November 1990) . 

126 Janet Jewett, "School-Based Early Childhood Centers," 
Northwest Regional Educational Laboratory (July 1991), p. 10. 

127 Tommy M. Tonlinson, "Class Size and Public Policy: 
Politics and Panaceas," (Washington, DC: U.S. Government Printing 
Office, 1988). For additional studies of the complexity of the 
relationship between class size and student achievement, see 
Douglas E. Mitchell, Sara Ann Beach, and Gary Badarak, "Modeling 
the Relationship Between Achievement and Class Size: A Re-Analysis 
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Overall, the quality of research, evaluation, and development 

produced by the labs varies greatly — even more so than among the 

R&D centers. Contrary to the statements of some policy makers , 

some of the work of the labs is of high quality and usefulness. 

Most of the materials from the Far West Lab, for example, are as 

well-done conceptually and statistically as the products from most 

of the R&D centers. Yet there are serious weaknesses in the 

quality of much of the research put out by the other labs. While 

all of the labs have at least some staff who appreciate high 

quality research and know how to produce it, not everyone at those 

institutions has a strong commitment to the type of rigor and 

expensive methodology necessary for improving the quality of 

12 8 

materials produced under task 3. 

We have spent more than $800 million (in constant 1982-84 
dollars) on the labs since 1966. A sizable proportion of this 
money has been spent on research and development. Of the funds 
dedicated to research and development, much of it has not been well 
spent — especially during the past 10-15 years when applied research 



of the Tennessee Project STAR Data," California Educational 
Research Cooperative, University of California, Riverside (October 
1991); Mitchell, Carson, and Badarak, "How Changing Class Size 
Affects Classrooms and Students"? Tommy M. Tomlinson, "Class Size 
and Public Policy: The Plot Thickens," Contemporary Education . 62 
(Fall 1990), 1, 17-23? Elizabeth Word, Charles M. Achilles, Helen 
Bain, John Folger, John Johnston, and Nan Lintz, "Project STAR 
Final Executive Summary: Kindergarten Through Third Grade Results 
(1985-89)," Contemporary Education . 62 (Fall 1990), 1, 13-16. 

128 Moreover, one of the real dangers is that many of those at 
the labs who understand and support more systematic research and 
development are among those who worked at the labs in the late- 
1960s and 1970s. As they retire, much of the commitment and 
expertise in the labs to research and development will be lost. 
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and development appears to have decreased in both quantity and 
quality. The original and useful vision of the lab founders has 
not been fulfilled as many of the labs have become mainly technical 
assistance providers for their regional clients. Given the limited 
amount of original applied research or development done by the labs 
as well as the uneven quality of it, OERI's continued, substantial 
investment in this area needs to be reconsidered thoroughly and 
carefully. 

Of course, much ol the lab expenditures are for activities 
other than applied research or development — especially in more 
recent years. This analysis did not address the quality or the 
efficiency of those other activities. Therefore, while much of the 
applied research or development produced by the labs has been 
disappointing, this does not necessarily mean that the work of the 
labs overall has been problematic. Naturally, a detailed and 
careful assessment of the other lab activities (such as their 
provision of technical assistance) should be undertaken soon. 



D. Lab Review Process 

As we have already seen, the labs faced competition for the 
first time in 1985 and 1990. Although there had been concern about 
the process, by all accounts it was fair and non-controversial. 
Despite concerted efforts by the Office of Educational Research and 
Improvement (OERI) to stimulate competition for the lab cortracts, 
in fact most of the incumbents were unopposed in either 1985 or 
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1990. 

The lab competition for 1990 specified th^t separate awards 
would be made for each of the 10 regions designated (the 9 previous 
regions plus the addition of the Pacific Basic Region). Seven 
criteria were employed by the reviewers in making the awards: (1) 
regional needs (10 points) ; (2) relationships in region (20 
points) ; (3) institutional capability (15 points) ; (4) plan of 
operation (25 points) ; (5) key personnel (20 points) ; (6) 
evaluation plan (5 points) ; and (7) adequacy of resources (5 
points) • 

Unlike the criteria for the R&D centers, relatively little 

129 

emphasis was placed on the mission statement for the labs. 
More emphasis was placed on the importance of ties to the region 
and collaborations with other organizations. 

The discussion of the research and development activities 
listed a large number of acceptable deliverables including 
newsletters, policy papers, concept papers, research reports, 
resource guides, handbooks, bibliographies, occasional bulletins, 
instructional materials, technical papers, synthesis papers, 
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Under section c of the plan of operation, it did ask 
"[t]he degree to which the objectives of the project relate to the 
overall mission of the laboratory program and also special emphasis 
in this statement of work (at-risk students, rural small schools 
and early childhood education)." Office of Educational Research 
and Improvement, "Technical Evaluation Form: Regional Educational 
Laboratory Competition," (1990), p. 6. But this was only one of 10 
subcategories under the plan of operation. 
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manuals, conference proceedings, and research instruments, 130 
But in the scoring sheets for the reviewers, there was almost no 
emphasis on the importance of a specific research design for the 
proposed projects. Under the plan of operation, one of the ten 
subcategories was the quality of the project design but this item 
does not seem to have received much notice or attention during the 
competition. 

Reading through the five 1985 and 1990 applications, one is 
struck by the lack of specificity for many of the proposed research 
or development projects. Most projects are described in general 
terms without specifying such important information as what types 
of instruments will be used to measure student outcomes or the size 
of the samples to be drawn. Perhaps this lack of detail about the 
research and development reflected in part the OERI guidelines to 
the labs which called for only short-term research and development 
projects and did not require a detailed research design. But this 
lack of specific details about the proposed activities was also a 
more general characteristic of the response of the applicants to 
the other questions as well. 

Since most of these labs did not face any real competition, 
many of the reviewers chose not to make elaborate comments about 
the nature or quality of the proposed reserrch and development 
activities. In the case of two regions, however, the proposals 
were deemed so weak that it became necessary for the labs to 
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13 Office of Educational Research and Improvement, "Regional 
Educational Laboratory Request for Proposal, 1990," (Washington, 
DC: Office of Educational Research and Improvement, 1990), p. 52. 
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resubmit their revised applications twice before they were judged 
minimally acceptable. 

Thus, while the lab competitions were fair and well-handled, 
little attention was paid to the nature or quality of the proposed 
research or development activities at that time* A few reviewers 
did question the lack of specificity of the plans, but most 
accepted the original or revised lab submissions without too much 
protest. While the OERI staff triei to improve the overall quality 
of the lab proposals, in several instances it was difficult to 
overcome the lack of a carefully specified research design or of a 
detailed development plan. 

In both the 1985 and 1990 lab competitions, applicants were 

required to develop their own self -evaluation plan. For example, 

in the Regional Educational Laboratory Request for Proposals in 

1990, task 5 was to conduct an evaluation of their own activities. 

Labs were expected to do both formative and summative evaluations. 

Among the four questions raised in the formative evaluation 

section, one explicitly addressed the issue of the adequacy of the 

research design: H Do applied research projects have appropriate 

13 1 

designs that are carefully implemented?" The quality of 

research issue received further reinforcement and clarification 
when the proposal stated that H [t]he contractor shall develop and 
implement a plan to assure high quality of its deliverables and R&D 



Office of Educational Research and Improvement, "Regional 
Educational Laboratory Request for Proposal, 1990," p. 29. 
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132 

products and publications." 

Unfortunately, the results of the lab self -evaluations and 
quality assurance have been very mixed. The Far West Lab generally 
has done a good job of quality assurance and is developing a 
critical and useful system of reviewing its activities . Evaluation 
of products and activities is done by each individual program or 
project , but the Director of Planning and Evaluation reviews their 
evaluation plans and reports as well as provides technical 

assistance to them. A sample of the lab f s publications are sent 

13 3 

out for external review. While some improvements might be 

made in their impact analysis , on the whole the Far West seems to 
be coping successfully with the requirement to monitor and assess 
the quality of its products and services. 

At some of the othar labs, however, the self -evaluation 
activities have been less successful. Much of their evaluation 
activities have been devoted to an analysis of client satisfaction- 
-a legitimate concern and one of the four formative evaluation 
questions asked in the 1990 lab RFP. But much less attention has 
been paid to a critical assessment of the quality of the applied 
research or development projects. Moreover, some of the labs did 
not always submit their research syntheses or policy papers to the 
scrutiny of other experts in the field. As a result, while OERI 



9 

ERLC 



132 Office of Educational Research and Improvement, "Regional 
Educational Laboratory Request for Proposal, 1990," p. 31. 

133 Paul D„ Hood, "Responding to Educational Needs and 
Opportunities for School Improvement in the Far West: The Fourth 
Annual Self Evaluation Report of Far West Laboratory Regional 
Programs" (January 1990) . 
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had intended the lab self -evaluation activities to play an 
important role in maintaining the quality of the services and 
deliverables produced by the labs, in practice the results were 
often disappointing. 



E. Lab Monitoring 



Programs for the Improvement of Practice (PIP) under the 
direction of Eve Either is one of the major offices within the 
Office of Educational Research and Improvement (OERI) and has the 
responsibility for overseeing the activities of the labs. The 
Educational Networks Division (END) under the direction of David 
Mack is one of the three divisions within PIP and administers the 
Regional Educational Laboratory Program. 

Six programs, including the labs, are administered within END. 
The labs are directly overseen by the Laboratory Team under the 
direction of Charles Stalford. 134 Each lab is assigned a 
separate institutional liaison. 135 The institutional liaison 



Near the end of this investigation, Charles Stalford has 
been given the assignment to direct the Eisenhower Math and Science 
Program within OERI. He has been replaced by Marshall Sashkin, the 
institutional liaison for the Far West Lab. 

135 since laboratories are institutional programs rather than 
just projects, the term "institutional liaison" is used. 
Educational Networks Division, "Handbook for Institutional 
Liaisons," Office of Educational Research and Improvement (April 
1992). Since the handbook is a loose leaf manuscript, it will not 
be possible to cite the appropriate pages when this document is 
being quoted. 

Diane Ravitch, the Assistant Secretary for OERI, strongly 
objected to the word "liaison" for designating the staff members 
overseeing the R&D centers and preferred instead the word 



179 

has the responsibility for overseeing the activities of his or her 
lab and coordinating those activities with other members of the 
Laboratory Team. While each institutional liaison has many other 
responsibilities within PIP, their work with the labs is seen as 
their primary responsibility. 

As we have seen in the previous chapter, there was some 
confusion among the center monitors on the exact nature of their 
jobs as well as on the rules and regulations that governed the R&D 
centers. A partial explanation for the confusion among the nearly 
two dozen center, monitors was that they were subdivided within the 
Office of Research (OR) into four different divisions which had 
different philosophies about the role of a center monitor. 
Moreover, the center monitors rarely met together as a group to 
discuss common problems and how to interpret the rules and 
regulations governing their grantees. The Office of Research (OR) 
at that time also did not have a handbook for monitors which might 
have helped to standardize their responses and practices. 

The activities and interpretations of the rules and 
regulations of the lab institutional liaisons appear to be more co- 
ordinated — despite the fact that the individual labs exhibit 
considerably more diversity amongst themselves than the R&D 
centers. A major reason for the co-ordination among the 



"monitor." She felt that the term "monitor" implied a much more 
active oversight of the R&D centers than the word "liaison." As a 
result, the Office of Research (OR) now calls its staff members 
overseeing the R&D centers "monitors." Apparently, Ravitch has not 
made the same point about the individuals dealing with the labs so 
they continued to be called "institutional liaisons." 
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institutional liaisons is that all members of the laboratory team 
meet together for several hours each week to discuss lab policies 
and practices. This gives them a\n opportunity to get advice from 
each other as well as to discuss their interpretations of the rules 
and regulations governing the labs. Moreover, the Educational 
Networks Division (END) recently has developed a handbook for the 
institutional liaisons which should be useful for instructing new 

members of the Laboratory Team as well as for helping to 

136 

standardize practices and interpretations. 

As among the center monitors , there was considerable diversity 
in the background and experiences of the lab institutional 
liaisons. Giver, the greater range of different activities 
performed by each of the labs than by each of the R&D centers , it 
may be more difficult for individual institutional liaisons to 
effectively monitor their entity than for the center monitors. 
This problem has been recognized by the Laboratory Team as it 

recommends "a team approach to monitoring the quality and 

137 

usefulness of the program activity of the laboratories...." 

The institutional liaisons vary considerably in their ability 
to evaluate lab research and development activities. A few are 
well-prepared to assess the research design and statistical 
procedures used by their labs, but most of them do not have the 



Educational Networks Division, "Handbook for Institutional 
Liaisons." The handbook has been placed in loose-leaf binders so 
that it can be easily updated to reflect any new regulations or 
activities. 

137 Educational Networks Division, "Handbook for Institutional 
Liaisons. 11 
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training and research experience necessary for that particular 
task. 138 Many of the institutional liaisons have sought 
assistance from others within OERI to help them evaluate some of 
the research and development work performed as part of the lab's 
task three assignment. In the past, there was an evaluation unit 
within END that assisted lab liaisons. Unfortunately, that unit 
has been eliminated so that the lab liaisons now find it more 
difficult to receive the assistance they need in this area. 

In theory, the R&D centers and labs are administered quite 
differently as the former are funded either as grants or co- 
operative agreements while the latter are funded as contracts. 
Presumably government contracts could be more closely monitored by 
the OERI staff than either grants or co-operative agreements, but 
in practice the reverse appears to be true. Probably the major 
explanation for this seeming discrepancy is that the governance and 
direction for the labs has been increasingly left to their regional 
boards. As a result, even though OERI has contracts with the labs, 
many of the institutional liaisons feel uncertain or ambivalent 
about their authority over these institutions. Most of the lab 
institutional liaisons appear to feel less able to direct and guide 
the activities cf their institutions than their center monitor 
counterparts . 

Most center monitors are able to visit their institutions only 
once a year, but the lab institutional liaisons are expected to 



138 About half of the members of the lab team have an advanced 
degree beyond a BA. But few of the lab team have had much 
opportunity in recent years to be active researchers themselves. 
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conduct two site visits* In addition, the lab directors and some 
of their staff are more likely to meet with representatives from 
OERI in Washington, DC than are the center directors and their 
staff. The more prevalent physical contact between the labs and 
OERI staff may help some of the institutional liaisons to be more 
knowledgeable about certain aspects of their contractors than their 
colleagues monitoring the centers. 

The institutional liaisons for the labs examine the budgets 
submitted by their contractors, but they do not seem to pay too 
much attention to how the money is spent substantively. The PMIS 
forms which might provide some guidance on where and how lab money 
is spent are generally ignored by both the labs and their 
institutional liaisons. While almost all of the labs do fill out 
their PMIS forms as required by the contract, some of them do it so 
casually and eclectically as to render the results suspect for any 
analytic or comparative purposes. Moreover, the manner in which 
many labs report their expenditures on the six tasks makes it very 
difficult for someone to ascertain how the money is expended by 
individual project. Many of the budgets for the labs are not 
sufficiently detailed to permit an analysis of expenditures by 
individual projects. Some of the institutional liaisons do not 
seem to pay much attention to the information from PMIS in 
analyzing how their lab is functioning. 

We noted the difficulties in getting center monitors to 
terminate the funds of individual research projects which were 
deemed unworthy and unsalvageable (and the money saved being either 

1 
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redistributed within the same center or to another R&D center) . 

The same seems to be true for the labs. Institutional liaisons 

appear to be very reluctant to stop the funding of individual 

projects within their labs — perhaps in part because of the inherent 

difficulties involved in evaluating the quality of the diverse 

services provided by these institutions. Most lab directors do not 

appear seriously concerned that OERI might decide to stop the 

funding of any of their individual projects. 

Perhaps the most surprising discovery was the lack of a 

detailed third-year review of the labs in 1992. OERI had done a 

thorough and careful third-year review of the labs in 1987 — 

including the use of outside teams of experts visiting those 

institutions. Those reviews had uncovered some important 

weaknesses in some of the individual labs and provided useful 

information for planning for the 1990 lab competitions. Indeed, 

compared to the third-year R&D center reviews for 1987, the ones 

139 

for the labs were much more systematic and rigorous. 

As we have seen in the previous chapter, the Office of 
Research (OR) conducted a thorough and detailed third-year review 
of the R&D centers in 1992. Part of the impetus for the review of 
the centers came from OR and part from Diane Ravitch, the recent 
former Assistant Secretary of OERI, who was anxious to investigate 
and improve the quality of research being done by the R&D centers. 
One might have thought that a comparable third-year review of the 

139 Unfortunately, the third-year review in 1987 did not look 
at the quality of the work being produced by the labs. 
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labs would have been undertaken simultaneously — especially since 

several of the labs funded in 1990 did not receive high marks from 

the original reviewers. 

There are several explanations for the decision not to subject 

the labs to a third-year review in 1992. The Grants and Contracts 

(OGC) Office in the U.S. Department of Education did not encourage 

a third-year review because they interpreted the existing contract 

140 

as a five-year package that could not be renegotiated. Some 

of the leadership in PIP also wondered if a third-year review might 

be too expensive to conduct if teams of outside reviewers were used 

as in 1987. Moreover, there does not appear to have been any great 

concern within PIP that the quality of the products and services 

being produced by the labs were seriously deficient and therefore 

needing closer scrutiny. Perhaps the growing sense that the labs 

were being guided and monitored more by themselves than by the 

institutional liaisons militated also against any desire for a 

141 

systematic third-year review of these institutions. Finally, 
with the energy of the Office of the Assistant Secretary focused on 
the third-year review of the R&D centers, little attention was paid 
to the fact that PIP had decided not to do a third-year review of 
the labs. 

Whatever the reasons for the decision not to conduct a 
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In 1987 the Grants and Contracts (OGC) Office did allow 
the five-year lab contracts to be subdivided into two components. 
However, the same interpretation was not given in 1992 — perhaps in 
part because of s/'caff changes in that unit. 

141 PIP did initiate an analysis of the governance of the labs 
through a three-year contract with Policy Studies Associates. 
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thorough third-year review of the labs, that decision was 
unfortunate. The uneven quantity and quality of research and 
development work revealed in this investigation of five of the labs 
suggests that a more careful and thorough review of all of the 
activities of the 10 labs probably would have been useful. 
Moreover, given the fact that the National Academy of Science 
analysis of OERI did not consider in any detail the functioning of 
the R&D centers or the labs, an OERI -sponsored third-year lab 
review might have been timely both for improving the functioning of 
the current labs as well as for preparing for the recompetition of 
those institutions in 1995. 

Given the explicit requirement that the labs conduct their own 
self -evaluations of the quality of their service deliverables, the 
institutional liaisons may need to pay closer attention to these 
efforts. If the lab self-evaluation task was fully and 
satisfactorily implemented, much of the quality assurance work of 
the institutional liaisons would be done for them. Then the 
institutional liaisons could rely more on the lab's own evaluation 
system and check periodically to make certain that it was 
functioning effectively to guarantee the quality of the work being 
performed by those institutions. 

Some analysts would emphasize the importance of improving lab 
self -evaluations of quality even further. Indeed, a few 
knowledgeable and thoughtful employees in PIP even have questioned 
the wisdom of focusing on the role of the individual institutional 
liaisons for enhancing quality of the research and development 
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products; rather, they would rely on use of total quality 
management (TQM) at the labs to improve and ensure the quality of 
their research and development work: 



[W]e take issue with the [draft] report 1 s attribution of such 
problems to the quality and abilities of the institutional 
liaisons. The fact is that the form of monitoring advocated 
in the report would not and cannot lead to improved quality of 
products. We do not have here the space required to explain 
what Deming, Juran, and other international quality experts 
have been teaching (mostly to managers and executives in 
organizations in Japan and elsewhere outside the U.S.). To 
try to sum it up as succinctly as possible (and more than a 
little over-simplistically) quality is achieved only if it is 
designed into the system; it cannot be produced by inspection 
of product (though review and inspection is not 
irrelevant) . 

While everyone can agree that the responsibility for 
developing and employing appropriate mechanisms for ensuring the 
quality of lab research and development ultimately begins with 
those institutions, it is less evident that many, if not most, of 
the labs will develop and use such a system based upon the TQM 
principles — especially without active and ongoing prodding by the 



142 Comments from some employees in PIP on the earlier draft 
report (March 25, 1993). 
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OERI institutional liaisons. As discussed earlier, the Far West 
Lab certainly has developed and administered an effective self- 
monitoring system for ensuring the high quality of the xr products. 
But most of the other labs investigated in this study had not. 
Given the usually high, additional costs of improving research and 
development, there are few tangible incentives for labs to devote 
more of their scarce resources to upgrade the quality of the 
research and development — particularly if OERI and their other 
clients continue to accept the lower-cost and lower-quality R&D 
products without too much question. For TQM to work effectively, 
management first must accept the need for such changes and 

improvements and then be willing to invest the necessary resources 

143 

for creating and supporting such a culture. Therefore, while 

OERI needs to work with the labs to improve their self -evaluation 
systems, the institutional liaisons will need to continue to 
monitor the quality of the research and development produced by the 
labs. 

Overall, one has the impression that the Laboratory Team in 
OERI is composed of some very capable individuals who pursue their 
responsibilities earnestly and diligently. Moreover, the 
organization and functioning of the laboratory team as a group is 
exemplary. Yet the ambiguities of the relationship of the labs to 
OERI are sufficiently great so that the institutional liaisons are 
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For a useful and thoughtful discussion of TQM and it 
limitations, see Marshall Sashkin and J. Kiser, Putting Total 
Quality Management to Work (San Francisco: Berrett-Koehler 
Publishers, 1993) . 
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not exercising the type of leadership and active monitoring that is 
needed given the uneven quality of research and development 
produced by the labs, 

III. Conclusion 

When the education laboratories were created in the mid-1960s , 
it was envisioned that they would be large-scale, national 
institutions devoted to systematically developing educational 
materials based upon research. But in practice that model of the 
labs was challenged from the very beginning as the U.S. Office of 
Research created a large number of smaller and more regional 
institutions. Nevertheless, the vision of large-scale labs which 
systematically developed curriculum and other education products 
survived through much of the 1970s both among policy makers in 
Washington, DC as well as among many of the lab directors. 

In the late 1970s, however, both the vision and the reality of 
the labs were altered significantly — in large measure due to the 
pressures from the U.S. Congress at the behest of the labs. 
Increasingly labs became portrayed as regional entities whose 
primary function was to provide research-based technical assistance 
for their local constituents . This trend was reinforced and 
consolidated during the 1980s under the Reagan and Bush 
Administrations. An outright prohibition was placed on the 
development of large-scale curriculum and even the amount of 
original short-terra applied research being done at the labs appears 
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to have been reduced. The growing power of the regional labs' 
governing boards came at the expense of control and direction for 
these institutions from the Office of Educational Research and 
Improvement (OERI) . Today, for all practical purposes, many of 
the labs are primarily regional institutions offering research- 
basea technical assistance and governed mainly by their own 
regional boards. 144 Some of the labs, like the Far West Lab, 
continue to sponsor a significant amount of high quality short-term 
applied research and even a smaller number, like the Mid-continent 
Lab, still try to develop their own curriculum packages— though 
usually with non-OERI funds. At the same time, there is now a 
welcome trend toward cooperation and coordination among the labs on 
specific topics such as, the improvement of math and science in the 
local schools; the revitalization of small, rural schools; and 
facilitation of the transition from preschool education to the 
elementary grades. 

Taking into consideration the large changes in the mission and 
operation of the labs over time as well as the considerable 
diversity among them today, now would be an opportune time for OERI 
and others to rethink thoroughly the roles of the labs— especially 
given the recent efforts within OERI to develop a 3-5 year- 
strategic plan, the need to recompete the labs in 1995, the 
impending reauthorization of the agency, and the change in 
administrations. How one defines and interprets the future mission 

144 As the recent National Academy of Science report put it, 
"...the laboratories are now primarily service providers." 
Atkinson and Jackson, eds., Research and Reform, p. 78. 
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of the labs will have a profound effect on how research and 

development are viewed and executed in this agency. 

One might argue that the labs should basically remain as 

regional institutions providing research-based technical assistance 

to state education institutions and local school districts. This 

is certainly a plausible and defensible position. However, it also 

may have some long-term implications for the labs and OERI that 

need to be carefully explored and considered. For example, just 

how many different institutions need to be funded by the U.S. 

Department of Education to provide technical assistance? According 

to one estimate, the U.S. Department of Education is already 

funding 43 different programs to provida technical 

assistance. 145 An OERI survey of the U.S. Department of 

Education identified 12 separate technical assistant programs 

legislatively mandated which funded 135 projects or institutions at 

14 6 

an annual cost of $90 million. Would it be more cost- 

effective to consolidate some of these providers to minimize 
unnecessary duplication and to make it easier for local schools to 
obtain assistance? Would it be more efficient simply to provide 
directly to the states the federal money now allocated for training 
and technical assistance in order to let them purchase whatever 
particular services they need? As the regional education labs in 



* M. Bruce Nestlehut, "Management Improvement Report, No. 
91-10" (Atlanta, GA: Office of Inspector General, U.S. Department 
of Education, 1991), pp. 1-2. 

146 Atkinson and Jackson, eds. , Research and Education 
Reforms , p. 122. 
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effect have to compete against other types of providers of 
technical assistance, will the quality and cost of their services 
be competitive? 

Another possible complication of having the labs focus on 
research-based technical assistance to the regions may be the 
difficulty of maintaining the quality of the research staff 
necessary to carry out these functions. As the research and 
development professionals at the labs find little opportunity to 
pursue any original work, will the best ones be willing to stay and 
provide the needed expertise to evaluate and synthesize research 
and development materials produced elsewhere? And will the 
replacements for any of the departing research and development 
professionals have the necessary conceptual and technical skills to 
provide critical research-based assistance to the regional clients? 

Finally, given the limited amount of research and development 
funding within OERI, does it make sense to devote such a large 
proportion of its total budget to labs which are only providing 
research-based technical assistance— especially if other units 
within the U.S. Department of Education are already providing 
substantial funds for technical assistance? The assumption in some 
of the recent OERI publications about the labs is "that enough is 
known from research and development and successful practice to help 
students and schools...." 147 While it is true that we do know 
enough about some education problems and solutions from research 



147 Educational Networks Division, "Handbook for Institutional 
Liaisons." 
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and development to disseminate that information to practitioners, 
unfortunately there is even much more that we do not know. 
Particularly missing at OERI today is the systematic development of 
educational products and services — an essential link between 
research and the teachers and administrators needing that 
information. Therefore, is it wise to focus the efforts of the 
labs mainly on regional dissemination of information and on the 
provision of technical assistance? 

An alternative scenario is to expand the role for systematic 
applied research and development at the labs much along the lines 
envisioned for those institutions in the 1960s and early 1970s. 
While much of the regional services of the labs could be 
maintained, there would be greater emphasis on doing original 
applied research and development which would be useful not just to 
the lab's regional clients, but to the nation as a whole. Indeed, 
the need to make the findings from the lab investigations more 
generalizable and useful to others outside the region is already 
stated in the existing guidelines for the design of applied R&D lab 
projects. 148 Moreover, a substantial amount of funds is already 
being allocated for task 3 — applied research and development. More 
of those task 3 funds might be used for original applied and 
development research rather than for producing research syntheses 
or policy papers (some of which might be done nationally rather 
than individually for each of the 10 separate labs) . 



Educational Networks Division, "Handbook for Institutional 
Liaisons. 11 
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Interestingly, this vision of labs which combines both national and 

regional concerns was one shared by many individuals within NIE and 

149 

CEDaR in the mid-1970s. 

The total amount of spending in real dollars has decreased 
over time for the labs . Since many of the 1 2bs are now 
concentrating on providing small-scale services to clients anyway, 
the diminished size of the labs over time does not appear to have 
been the major factor in inhibiting more systematic and rigorous 
research and development. If the labs do begin to engage in more 
large-scale and long-term systematic development activities, 

however, then it may be necessary at some point to reconsider the 

150 

size of their overall budget. Initially, much of the 

additional funds for more systematic and large-scale applied 
research or development projects, however, can come from a 
redirection of existing monies rather than the addition of new 



While the National Academy of Science panel did not stress 
the need for more applied and development work at the labs in their 
recommendations, they did suggest that the "innovative methods, 
programs, and processes [produced by the labs] should be subject to 
a quality assurance review — such as those conducted by the Program 
Evaluation Panel — before wide-scale distribution. 11 Richard C. 
Atkinson and Gregg B. Jackson, eds., Research and Education Reform: 
Roles for the Office of Educational Research and Im provement 
(Washington, DCs National Academy Press, 1992), p. 152. 

150 The National Academy of Science analysis of the labs 
focused mainly on an expansion of staff as state liaisons at an 
additional annual cost of approximately $5 million to OERI. The 
panel did not call for an increase in overall lab funding — perhaps 
in part because they did not stress an expanded role for applied 
research and development in those institutions. Atkinson and 
Jackson, eds., Research and Educ ation Reform, pp. 151-154. 
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funds . 

As we have said in regard to the R&D centers, if there is no 
substantial increase in funding for mission-oriented or field- 
initiated research, it may be prudent to reallocate monies from the 
existing OERI funds for the R&D centers and labs. Again, perhaps 
10 percent of the current lab budget should be redirected to 
mission-oriented or field-initiated research in order to have a 

more balanced portfolio of research and development activities as 

152 

the National Academy of Science study of OERI recommended. 

In our earlier analysis of the R&D centers, we lamented the 
absence of basic research and suggested how it might be enhanced. 
Certainly the labs should be allowed to pursue some laiss^ on- 
oriented basic research as well. But a more important area for 
expanded activity for the labs would be to focus more on the 
systematic development of curriculum and other educational 
products. It is also time to revisit the long-standing outright 
prohibition against any OERI funding of large-scale curriculum 
development. While some of the cautions of the late 1970s and 



151 Several individuals within OERI were concerned that the 
renewed emphasis on development by the labs would be used only as 
a rationale for these institutions garnering an even larger 
proportion of the scare research dollars in the agency. Clearly, 
that is not the intent of this recommendation. Since much of the 
existing task 3 funds, for example, are not well spent, they should 
be reallocated toward more systematic development efforts. 
Moreover, the recommendation for spending more funds on development 
is based upon the assumption that they will be devoted to 
systematic development of educational products and practices. 
While one can legitimately define development in much broader 
terms, the intent of th: s recommendation is restricted to a 
narrower definition which focuses on systematic development. 

152 Atkinson and Jackson, eds., Research and Education Reform. 
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1980s against such investments must be considered and addressed, it 

seems somewhat incongruous to continue such a blanket prohibition 

at a time when other agencies such as the National Science 

Foundation are making substantial federal investments in curriculum 

153 

development for science and math education. 

Whether the funds allocated for more systematic development 
should be given directly to each lab from their existing OERI 
appropriations or pooled together and bid upon by the labs and 
other institutions such as the R&D centers needs to be explored 
further. One of the major advantages of having some type of 
competition for long-term developmental funds is that it would help 
OERI and its reviewer panels to select the strongest proposals from 
those institutions which are really interested in participating in 
such endeavors. 

If the mission of the labs is reordered to emphasize more 
large-scale and systematic development, then obviously it may be 
necessary to redirect some of the existing development funds from 
some of the small and short-term projects the labs are currently 
pursuing. Moreover, given the unequal quality of applied and 
developmental research that has been encountered in this 
investigation, more emphasis needs to be placed on the quality of 
the work produced. Some of the current lab evaluation plans may 



±i} ^ Perhaps one could adapt some of the safeguards that the 
NSF has introduced in its regulation of curriculum development in 
recent years to avoid many of the criticisms it received for its 
MACOS project. For a helpful discussion of that controversy at NSF 
and the attempts by that agency to rectify the situation, see Dow, 
Schoolhouse Politics . 
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need to toe strengthened by the inclusion of more outside reviewers 
with the requisite social science skills necessary to assess the 
various types of applied research and systematic developmental 
efforts that the labs may be pursuing. 

If the labs move toward the production of more systematic and 
large-scale applied research and development, it may be useful for 
them to increase the representation of individuals on their 
regional governing boards who are experienced and trained in 
research and developmental tasks. At the present time, with the 
exception of a few labs, it does not appear that there are many 
persons on the existing regional governing boards who are 
experienced researchers or well-trained developmental ists. By 
including more researchers and developmental ists, it will provide 
the regional governing boards with the expertise necessary to guide 
and monitor the work of those institutions; it will also reinforce 
the value of such activities among the regional governing boards. 

The quality of the research and development at the labs can 
also be enhanced by several improvements at OERI. The next 
competition for labs in 1995 should request more specific 
information about the design and operation of the proposed applied 
research and development projects. The technical panels assigned 
to evaluate those proposals should have sufficient representation 
from individuals expert in research and development in order to 
assure a high level of quality. Again, as with the reviewers for 
the R&D centers, they should be modestly compensated in order to 
ensure being able to recruit the most competent and appropriate 
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individuals. 

The limited analysis of the labs by the National Academy of 
Science panel led them to question the advisability of any future 
competitions for those institutions due to the lack of competition 
and to the concern that the process did not improve the performance 
of the incumbents. 154 While the panel has raised a legitimate 
and important issue, unfortunately they did not conduct the type of 
careful and thorough review of the labs to warrant such a strong 
conclusion. My own perusal of the applications for the five labs, 
including one which had to resubmit their proposal twice before 
being funded, suggests that the application process may have 
provided a convenient and useful opportunity for OERI and the 
outside reviewers to clarify and influence the direction and nature 
of the proposed activities at those institutions. Given the 
continued difficulties for some OERI institutional liaisons to 
monitor and improve the operation of the labs today, the 
elimination of any future lab competition what-so-ever may be 
counter-productive. Moreover, before relying only upon the promise 
of a periodic review of the labs to monitor their progress, perhaps 
first we should be certain that OERI can and will develop an 
appropriate and effective lab x-eview system which ensures the 

154 Atkinson and Jackson, eds., Research and Educ ation Reform, 
p. 152. In subsequent public and private discussions with some 
individuals from the National Academy of Science panel, they 
stressed the high cost of running such competitions. However, if 
this was the basis of their decision, they might have provided us 
with some estimates of those costs. Moreover, given the sizable 
expenditure that the federal government invests in the labs over a 
five-year contract, would a small percentage of those funds 
expended on a useful competition really be inappropriate? 
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quality of the products being produced. Finally, as the mission 

and the orientation of the labs may change over time (as they have 

so frequently done in the past) , limiting the candidates for those 

new or revised tasks only to the current labs may be short-sighted. 

Thus, before considering any changes in the present system of 

holding competitions for the labs every five years, we may need a 

more indepth and systematic analysis of those institutions than the 
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National Academy of Science panel was able to provide. 

OERI must take a much more active role in guiding and 
monitoring the labs. While the agency should consult and work with 
the regional governing boards, the overall national directions for 
the labs must come from OERI — based, of course, upon careful and 
extensive consultation with local and^ state education officials, 
teachers, and parents. The recent efforts of OERI and the lab 
directors to develop a national system of regional educational labs 
is a welcome development and not inconsistent with the continued 
attention of the labs to the particular needs of their regional 
constituents. One of the important advantages of focusing on the 



There is a widespread undercurrent among some analysts and 
policy makers that the presence of an incumbent lab director on the 
National Academy of Science panel led to the recommendation that 
the labs should not have to face recompetition in the future. On 
the basis of my interviews some of the individuals connected with 
that effort, I did not detect any evidence to that effect. 
However, one might question the wisdom of the National Academy of 
Science for putting an active member from an OERI-funded lab and a 
R&D center on the panel since this might have inadvertently created 
the appearance of a potential conflict of interest. Indeed, one 
knowledgeable observer of the National Academy of Science study 
believes that the presence of a lab and a center director on the 
panel discouraged more open and candid discussions of the strengths 
and weaknesses of those institutions. 
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national character of the regional education lab system is that it 
permits a greater specialization of the applied research and 
development among the labs and the elimination of unnecessary 
duplication and waste. Moreover, this specialization might allow 
individual labs t o undertake some of the more large-scale and long- 
term applied research and development without diverting too much of 
the funds for providing technical assistance to regional clients. 

As the applied research and development products of the labs 
are oriented more to a national audience , it will also be 
appropriate and important to require that any of the quantitative 
research and development data assembled by those institutions using 
OERI funds be made available to others through a public use data 
archive. Again, as with the situation of the R&D centers, perhaps 
the guidelines for depositing data adopted by the National Science 
Foundation for its social science grantees could be used as the 
basis for developing a comparable policy for OERI. 

The Educational Networks Division (END) within the Programs 
for the Improvement of Practice (PIP) needs to monitor more closely 
the quality of applied research and development produced by the 
labs. More attention needs to be paid to the required lab self- 
evaluations to ensure that quality of their services and 
deliverables meet the highest standards. An indepth third-year 
review of all of the labs products and services would have been 
useful — comparable, but not entirely identical, to the review of 
the R&D centers by the Office of Research (OR) . Moreover, some of 
the institutional liaisons need to take a more active role in 
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monitoring the quality of the applied research and development at 
their institutions. As with the R&D centers, in situations where 
an individual project at a lab is deemed inadequate conceptually 
and methodologically, that project should be redesigned 
satisfactorily or those funds redirected to other activities within 
that lab or reassigned to another lab. The problem is not that the 
institutional liaisons are advocates or defenders of the labs. On 
the contrary, most of them can and have been quite critical of 
aspects of the labs. However, many of them often feel too 
powerless in the present circumstances to assert the type of 
leadership needed to monitor effectively the quality of work being 
done at those institutions. 

As we have noted earlier, the Laboratory Team at END is 
already doing a good job of co-ordinating their activities and 
standardizing their interpretations of the federal rules and 
regulations governing the labs. They are to be commended and 
encouraged to continue their co-ordination and co-operation amongst 
themselves. The development of a handbook for institutional 
liaisons is also an important step forward. Perhaps in the near 
future it may be possible to expand the handbook to include some 
additional suggestions on how to monitor more effectively the 
quality of the applied research and development produced in the 
labs. Finally, the ability of the institutional liaisons to visit 
the labs at least twice a year should be maintained. 

Since some of the institutional liaisons, like their center 
monitor countexparts , have not had sufficient training and 
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experience in research and development to oversee properly those 
activities in the labs, it may be necessary and useful to draw upon 
the expertise of others in END or elsewhere in OERI. Some of the 
institutional liaisons also may need and want additional training 
in social science methodology as well as development practices to 
enhance their skills. Perhaps it is also time to re-establish an 
evaluation unit within END or PIP that could provide technical 
assistance to the institutional liaisons when it is needed. 

One of the difficulties encountered in evaluating the nature 
and cost of the applied research and development at the Jabs was 
the inadequacies in the design and the reporting of substantive and 
budgetary information in the current PMIS system at OERI. The 
categories for the PMIS system should be expanded and defined more 
carefully in order to reflect more accurately lab activities and 
expenditures. Particularly, more attention needs to be devoted to 
the definition of development and it may be necessary to subdivide 
that broad category in order for everyone to be able to communicate 
more effectively with each other. The PMIS forms should be 
promptly and completely filled out by the labs and that information 
used by the institutional liaisons and other policy makers at OERI 
to analyze the activities of those units. 

It has been all too easy to forget the importance of worrying 
about quality in the research and development produced by the labs 
and the R&D centers. Neither OERI or the U.S. Congress has paid 
sufficient attention to it in the past, despite the occasional and 
prophetic warnings from individuals like Sam Sieber nearly two 
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In the context of current pressures for dissemination , it 

is easy to forget that not all information or innovations are 

really worth disseminating. Obviously, if information or 

innovations are unreliable, misleading, or unsuited to the 

situation, they can have repercussions which are altogether 

harmful. And even if not harmful, an accumulation of futile 

experiences with information or products might create an 

attitude of skepticism. toward R&D of all kinds. Further, even 

when an innovation is workable and effective, the o pportunity 

cost of this particular innovation rather than another might 

be undesirable. Although these points are obvious, it would 

« ■ 

seem that we are still wedded to the notion of "the more 

156 

utilization the better." 



Perhaps now is the time to set aside some of the past acrimony 
between OERI and the labs in order to work together to develop a 
truly effective and useful system for educational research and 
development which finally adequately takes into account the quality 
of the work being commissioned and produced. Otherwise we will be 
condemned to continue to waste much of the scarce federal resources 
we have for helping all of our children receive the high quality of 
education they need and deserve. 



Campbell et al., R&D Funding Policies of the National 
Institute of Education . p. 90. 
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Richard H. Lerner, Anne C. Petersen, and J. Brooks-Gunn (New York: Garland 
Publishing, 1991), II, 698-707. (with Susan M. Juster) 

"Changing Approaches to the Study of Family Life in the Past," in American Fa milies: 
A Reference Guide and Historical Handbook , eds. Joseph M. Hawes and 
Elizabeth I. Nybakken (Westport, Conn.: Greenwood Press, 1991), 15-32. 
(with Laura McCall) 

"Schooling and Poor Children "in 19th-century America," American Behavioral 
Scientist . 35, No. 3 (January/February 1992), 313-331. 

"History of Child Care in the United States Before 1950," in Child Care in 

Context: Cross-Cultural Perspectives , eds. Michael E. Lamb, Kathleen J. 
Sternberg, Carl-Philip Hwang, and Anders G. Broberg (Hillsdale, NJ: 
Lawrence Erlbaum, 1992), 185-206. (with Victoria Getis) 

"Historical Perspectives on Adolescent Pregnancy," Earlv Paren thood and Coming of 
Aae in the 1990s , eds. Margaret K. Rosenheim and Mark F. Testa (Rutgers, 
NJ: Rutgers University Press, 1992), 136-149. 

"Immigrants and Schooling in the United States," Immigrants in Two Democracies: 
French and American Experience , eds. Donald Horowitz and Gerard Noiriel 
(New York: New York University Press, 1992), 127-147. 

"Adolescent Pregnancy and Child Support," Children at Risk in America: History. 
Concepts, and Public Policy , ed. Roberta Wollons (Albany, NY: State 
University of New York Press, 1993), pp. 202-229. (with Lindsay 
Chase-Lansdale) 

"Impact of the Civil War on American Widowhood," in The Changing American 

Family: Sociological and Demogr a phic Perspectives , eds. Scott J. South and 
E. Stewart Tolnay (Boulder, CO: Westview Press, 1992), pp. 63-85. (with 
Amy Holmes) 

"From the Mathers to Madonna: Changing Attitudes toward Adolescent Pregnancy," 
in Adolescent Pregnancy , eds. Annette Lawson and Deborah Rhode 
(forthcoming), (with Susan Harari) 
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"An Historical Perspective on Support for Schooling by Different Age Cohorts," in 
The Changing Contract Across Generations , eds. Andrew Achenbaum and Vern 
Bengtson (New York: Aldine, 1993), pp. 45-65. 

"Early Childhood Education: Then and Now," Daedalus . CXXII, No. 1 (Winter 1993), 
151-176. 

"Death," in Encyclopedia nf American Social History, eds. Mary K. Cayton, Elliott 
J. Gorn, and Peter W. Williams, III (New York: Charles Scribner's Sons, 
1993), pp. 2063-2070. 

•Troubled Youth: An Analysis of Children at Risk in Early Modern England and 
Colonial and Nineteenth-Century America," in Adolescents at Risk* eds. 
Robert Ketterlinus and Michael Lamb (forthcoming), (with Gerald F. Moran) 

"Whose Responsibility? An Historical Analysis of the Changing Roles of Mothers, 
Fathers, and Society in Assuming Responsibility for U.S. Children, in 
Children and the Family Support Act , eds. Jeanne Brooks-Gunn and Lindsay 
Chase-Lansdale (forthcoming), (with Lindsay Chase-Lansdale) 

"Historical Perspectives on Parental Investment and Childbearing," in ftjrth 
Management , eds. Jane Lancaster and Michael Lamb (forthcoming). 

"Changing Perceptions and Treatment of Young Children in the United States," in 
Ima ges ofChildhood . eds. Michael E. Lamb, Philip Hwang, and Irving Sigel 
(forthcoming). 

"The Elderly and the Almshouses: Some Further Reflections, 8 in Social Structure 
and Aoino: Some Historical Perspectives , eds. Andrew Achenbaum and 
K. Warner Schaie (forthcoming). 

"School Readiness and Early Childhood Education: Some Historical Perspectives," 
in Historical Perspectives on the Cu rrent Education Reforms, eds. Diane 
Ravitch and Maris A. Vinovskis (forthcoming). 

Reports and Hearings : 

Task Force Director for drafting U.S., Congress, House Select Committee on 

Population, Report on Fertility and Contraception in the United State s, 
95th Congress, 2nd Session (December 1978), 128 pp. 

Task Force Director for hearings of U.S., Congress, House Select Committee on 
Population, Hearings on Fertility an d Contraception in the United States , 
95th Congress, 2nd Session (February- June 1978), 3 volumes. 

Assisted in supervising and editing of five more reports and eight additional 

volumes and hearings of the U.S. House Select Committee on Population, 1978. 

"Evaluation of the Activities of the Alan Guttmacher Institute for the Ford 
Foundation" (June 1980), 70 pp. 

"The Report of the Washington Area Second Team on Future Inadvertent Human Intrusion 
into the WIPP Repository" in Fx psrt Judgment on Inadvertent Human Int rusion 
Into the Waste Isolation Pilot Plant , eds. Stephen C. Hora, Detlof von 
Winterfeldt, and Kathleen M. Trauth (Albuquerque, NM: Sandia National 

FRir Laboratories, 1991), F1-F30. (with Theodore Glickman, Norman Rosenberg, and 

IJ^L Max Singer). 
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"Surveys of Academic Freedom Among Faculty, Students, and the Public: A Review of 
the Literature and Suggestions for Future Research" (September 1991) [for 
Office of Education Research and Improvement, Department of Education]. 



Review Essavs : 

"American Historical Demography: A Review Essay," Historical Methods 
Newsletter . IV, No. 4 (September 1971), 141-148. 

Review essay of Peter Knight's The Plain People of Boston. 1830-1860 in Joum?! 
of Interdisciplinary History . Ill, No. 4 (Spring 1973), 781-786. 

Review essay of Bernard Farber's Guardians o f Virtue: Salem Families in 1800 in 
The Family in Historical Perspective . No. 3 (Spring 1973), 2-5. 

"From Springfield to Omaha," Reviews in American History . I, No. 3 (September 
1973), 330-335. 

"The Demography of the Slave Population in Antebellum America: A Critique of 

Time on the Cross ." Jmirnal of Interdiscipli nary History. V, No. 3 (Winter 
1975), 459-467. 

"Estimating the Wealth of Americans on the Eve of the Revolution/ Journal Pf 
Economic History . XLI, No. 2 (June 1981), 415-420. 

"Cops and Corpses: New Approaches to the Study of Police and Crime in Nineteenth- 
Century America," Historical Methods . XVI, No. 1 (Winter 1983), 36-40. 

"Searching for Classes in Urban North America," Journal of Urban History. XI, 
No. 3 (May 1985), 353-360. 

"Responses to Comments of Michael Xatz and Edward Stevens on the Origins of Public 
High Schools ." History of Education Quarterly . 27, No. 2 (Summer 1987), 250- 
258. 

"Hasty Marriages or Hasty Conclusions," Public Interest . 90 (Winter 1988), 
128-132. (with Lindsay Chase-Lansdale) 

"Discussion of Recent Ph.D.s in American Economic History," Journal of Economic 
History . XLVIII, No. 2 (June 1988), 424-427. 

"The Darker Side of Educational Progress," Reviews in American History. XVI, 
No. 2 (June 1988), 255-259. 

"Stalking the Elusive Middle Class in Nineteenth Century America," Comparative. 
Studies in Society and History . XXXIII, No. 3 (July 1991), 



Short Book Reviews : 

American Historical Review . 1986, 1987, 1988, 1989 
American Journal of Education . 1989 

Annals of American Academy of Political and Social Science . 1989 
Business History Review . 1982 
^ Comparative Studies in Society and History . 1977 

Contemporary Sociology . 1984 ?9f" 
££IL Hastings Center Report . 1981 
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History of Education Quarterly . 1986, 1989 
ISIS , 1987 

Journal of American History . 1973, 1977, 1979, 1982, 1986 

Journal of Contemporary Sociology . 1984 

Journal of Economic History . 1970, 1986, 1991 

Journal of Interdisciplinary History . 1974, 1988, 1989, 1991 

Journal of Social History . 1982 

Journal of Southern History . 1983 

Michigan Academician . 1978 

Science . 1980 

Socio-Economic Studies . 1981 
William and Marv Law Review . 1969 



Scholastic Information : 

National Merit Scholarship to Wesleyan University, 1961-1965 

Graduated with Honors, Cum Laude, and Phi Beta Kappa from Wesleyan University, 
1965 

Dutcher Prize from Wesleyan University (excellence in history) 1965 

William Oay Leonard Award from Wesleyan University (all-around achievement), 1965 

University of Michigan Distinguished Faculty Award, 1988-1989 

Oanforth Fellowship to Harvard University, 1965-1969 

Fellowship to Cornell University Summer Program, 1967 

Research grants from Cambridge Project, Danforth Foundation, Harvard University 
History Department, Harvard Center for Population Studies, Milton Fund, and 
National Science Foundation, 1968-1972 ($11,450) 

Grant from U.S. Office of Education Research for project on Infant Education with 
Dean May ($1,250), 1971-1973 

Grant from Spencer Foundation ($2,100), 1973 

Fellowship from Rockefeller Foundation ($10,000), 1973-1974 

Grant from MIH for project on "Diagnosis and Treatment of Asylum Inmates from 
1818-1860" with Barbara Rosenkrantz ($150,000), 1974-1977 

Grant from Rackham School of Graduate Studies, University of Michigan, for study 
of "The Development and Functioning of Academic Departments" ($3,540), 
1975-1976 

Grant from the Rockefeller Foundation for project on "Voting Behavior in the 94th 
Congress on Population Policy" ($6,550), 1977-1978 
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Grant from NIE for project on "The Politics of Educational Reform in Nineteenth- 
Century Massachusetts: The Controversy over the Beverly High School in 
I960" ($12,831), 1979 

Fellowship from Social Science Research Council for Life-Span Summer Program 
($2,750), 1980 

Grant from the Ford Foundation for project on "The Politics of Abortion in the 
U.S.. Congress and the Nation, 1973-1980" ($32,080), 1980-1981 

Grant from NIA for project on "The Development of Retirement in Early America: 
An Analysis of Congregational and Presbyterian Ministers in New Hampshire 
from 1630 to 1875" ($14,229), 1980-1981 

Guggenheim Fellowship for project on "Death and Dying in Civil War America" 
($13,000), 1981-1982 

Grant from Rackham School of Graduate Studies, University of Michigan, for study 
of "Death and Dying in Civil War America" ($5,983), 1981-1982 

Grant from Spencer Foundation for Analysis of Education in Newburyport, 
Massachusetts in 1860 ($5,000), 1983 

Grant from Sloan Foundation for Conference on "New Directions for Demographic 
History" with Charles Tilly and Olivier Zunz ($9,900), 1986 

Fellowship to the Center for Advanced Study in the Behavioral Sciences, Stanford 
University, 1987-1988 postponed) 

Grant from NEH Summer Seminar Program for College Teachers ($71,133), 1987 

Grant from Spencer Foundation for "Determinants of High School Attendance in 
Antebellum America" ($7,500), 1987-1988 

Grant from Sloan Foundation for Summer Program in "Quantitative Methods and 
Reasoning for Historians" with Jerome Clubb ($64,110), 1987-1989 

Grant from American Institutions Program ($7,500) for research, 1988-1990 

Grant from University of Michigan ($7,500) for research, 1988-1990 

Grant from NEH Summer Seminar Program for College Teachers ($92,568), 1990 

Grant from Spencer Foundation for "The Crisis in Moral Education in Antebellum 
Massachusetts" ($7,500), 1991-1992 



Professional Activities: 

Presented 130 papers, lectures, or seminars at professional meetings, other 
colleges and universities, or before public groups. 

Commented on 23 professional papers. 

Chaired 10 scholarly sessions. 
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Presented material 16 times as member of panel. 

Helped organize workshop on Essex County [Mass.] History, June 1970 

Chairman of committee to organize a conference on Essex County History, Salem, 
June 1971 

Member of the Board of Editors of the Historical Methods Newsletter , 1972-1979 

Member of the Board of Editors of the American Quarterly. 1977-1980 

Member of the Board of Editors of the Journal of Family History. 1976— 

Member of the Board of Editors of Human Nature. 1989- 

Member of the Board of Editors of History of Education Quarterly. 1991— 

Member of the OAH Committee on Bibliography and Research, 1974-1977 

Co-organizer with Tamara Hareven of conference on "The Family in the Process of 
Urbanization" which was sponsored by the MSSB, Will lams town, Mass., July 
1974 

Consultant to NEH sponsored project under the direction of Richard Bushman to 
computerize the Massachusetts Valuation Lists of 1771, 1975-1976 

Member of the Publications Committee of the Social Science History Association, 
1976-1978 

Member of the Advisory Committee for Data Archiving Information on Aging, joint 
project of the Institute of Gerontology and Institute for Social Research, 
University of Michigan, 1978-1984 

Member of Local Arrangements Committee for the Social Science History Association 
meeting in Ann Arbor, Michigan, October 1977 

Testified on "Historical Perspectives on Abortion" at the hearings of Michigan 
Senate Committee on Health and Social Services, June 1979 

Taped half-hour TV interview with Harold Johnson of the University of Michigan 
Institute of Gerontology on "The Demography of Aging," August 1979 

Assisted in drafting staff working paper on legislative expenditures for the U.S. 
House of Representatives Select Committee on Committees, August 1979 

Guest editor of the Michigan Law Review . August 1979 

Chairman of the committee to select best articles from the Journal of Social 
History . 1979-1980 

Member of the Advisory Group on Child and Family Indicators of the Social 
Indicators Program of the Social Science Research Council, 1979-1981 

Member of program committee for Social Science History Association meeting at 
Rochester, New York, November 1980 
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Consultant to the Ford Foundation to evaluate the activities of the Alan 
Guttmacher Institute, 1980 

Member of committee to evaluate the History of Fducation Quarterly, 1980 

Consultant to Center for Population and Family Health, Columbia University, on 
program for "Preventing Adolescent Pregnancy: The Role of the Family," 
1981 

Consultant to the National Archives and Records Service on disposition of FBI 
records, March 1981 

Interviewed on the "Politics of Abortion," for CBS Sunday Morning News, June 1981 

Member of panel for reviewers for summer research grants for the National 
Endowment for the Humanities, February 1982 

Member of Committee on History for 1983 Society for Research on Child 
Development Program 

Member of Committee on Biosocial Life-Span Approaches to Parenting and Offspring 
Development, Social Science Research Council, 1983-1991 

Member of National Advisory Panel on The Data Archive on Adolescent Pregnancy and 
Pregnancy Prevention, 1983-1986 

Member of Committee on Child Development and Social Policy for the Society for 
Research in Child Development, 1983-1985; Acting Chairman, 1988 

Member of Panel to Select the Merle Curti Award for best book in American Social 
History for 1982-1983 

Member of Panel to Select the Bancroft Prizes for 1986 

Member of National Academy of Sciences Panel on The Problem of Teenage Pregnancy, 
1984-1986 

Instructor for Sessions on The Adolescent Family Life Program at SRCD Workshop, 
Cornell University, June 21-23, 1984 

Elected to Executive Committee of Social Science History Association, 1984-1987 

Member of Program Committee on Social Policy for SRCD meeting in Toronto, April 
1985 

Chairman, Committee on Research in Economic History,' 1984-1985 

Member of Advisor Panel, University of Wisconsin Press series on Life Course 
Development, 1985- - 

Member, AHA Committee on Quantitative History, 1984-1986; chairman, 1986 

Elected to Board of Directors of the History of Education Society, 1987-1989 

Project Coordinator, Quantitative Research in History: Historical Demography of 
the Nineteenth and Twentieth Centuries, U.S.-U.S.S.R. Exchange, 1985-1990 
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Discussed adolescent pregnancy on television NBC Today Show, April 1987; VIXYZ 
(Detroit), August 1987; WBZ TV (Boston), November 1987 

Discussed adolescent pregnancy on radio talk show, KING 1090, April 1987; WXYT 
1270, October 1987; WFAD, January 1988 

Member of Planning Committee for special issue of Daedalus on late adolescence 
and early adulthood, 1987 

Commentator on Best Dissertations in Economic History, 1987 

Member of Editorial Advisory Board for the American National Biography , 1987- 
1993 

Member of Advisory Board for Institute for American Values, 1988— 

Member of Committee to Select Director of the Gerald R. Ford Presidential 
Library, 1988 

Member of Editorial Advisory Board for the Encyclopedia of Adolescence, 1987- 
1990 

Member of SSHA Committee on Education, 1988-1990 

Member of American Antiquarian Society Committee on Education, 1988-1991 

Member of Social Policy Committee of the Society for Research on Adolescence, 
1989-1994 

Member of Committee on Interdisciplinary Affairs, SSRC, 1989-1991 

Chairman. Committee to .Select the Best Book in the History of Education for 1989- 
1990 

Member of Committee to Select Best Book on Children and Public Policy, 1989-1992 

Member of Planning Committee for special issue of Daedalus on children ages 
3-11, 1991 

Consultant to Sandia Laboratories on the WIPP nuclear waste disposal site in New 
Mexico, 1990-1992 

Member, Publications Committee, Social Science History Association, 1992-1994 

Member of Steering Committee, Center for Family Studies, University of Delaware, 
1989- 

Elected President of the History of Education Society, 1994-1995 
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OERI's Labs and Centers ( \ 

10 Regional Educational Laboratories 



The Office of Educational Research and 
Improvement (OERI) helps educators 
and policymakers solve education prob- 
lems in their schools through a network 
of 10 regional educational laboratories. 
Using the best available information and 
the experience and expertise of profes- 
sionals, the laboratories identify solu- 
tions to education problems, try new 
approaches, furnish research results 
and publications, and provide training to 
teachers and administrators. OERI re- 
cently awarded $1 62 million in contracts 
to operate the laboratories over the next 
5 years. As part of their individual re- 
gional programs, all laboratories will pay 
particular attention to the needs of at- 
risk students and small rural schools. 
The 10 laboratories are: 

Appalachia Educational 
Laboratory, Inc. (AEL) 

1031 Quarrier Street 
PC Box 1348 
Charleston, WV 25325 
(304) 347-0400 

(800) 624-9120 (outside West Virginia) 
(800) 344-6646 (in West Virginia) 
Board Chair Ralph Booher Jr. 
Executive Director: Terry L Bdell 
Deputy Executive Director: 

John R. Sanders 
Region Served: Kentucky. Tennessee, 

Virginia, and West Virginia 
OERI Liaison: Carol Mitchell 
(202) 219-2128 

Far West Laboratory for 
Educational Research and 
Development (FWL) 

730 Harrison Street 

San Francisco. CA 941 07-1242 

(415) 565-3000 

Board Chair: Joann Mortensen 

Executive Director: Dean H. Nafziger 

Region Served: Arizona. California, 

Nevada, and Utah 
OERI Liaison: Marshall Sashkln 
(202)219-2120 

Mid-continent Regional 
Educational Laboratory (McREL) 

Denver Office: 

2550 S. Parker R<±. Suite 500 
Aurora. CO 80014 
(303) 337-0990 
Kansas City Office: 
4709 Belleview Avenue 
Kansas City. MO 64112 



(816) 756-2401 
Board Chair William Soft 
Executive Director: C.L Hutchins 
Region Served: Colorado, Kansas, 

Nebraska, Missouri, Wyoming. North 

Dakota, and South Dakota 
OERI Liaison: Beverly E. Coleman 
(202) 219-2280 
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North Central Regional Educational 
Laboratory (NCREL) 

1900 Spring Rd. f Suite 300 
Oak Brook, IL 60521 
(708) 571-4700 
Boaud Chain Franklin Waiter 
Executive Director Jeri Nowakowski 
' Region Served: Minnesota, Wisconsin, 

Iowa, Illinois, Mchigan.tndiana, and 

Ohio 

OERI Liaison: Carter H. CcJIns 
(202) 219-2194 

Northwest Regional Educational 
Laboratory (NWREL) 

101 SW Main Street. Suite 500 
Portland, OR 97204-3297 
(503) 275-9500 
Board Chair: Barney Parker"" 
Executive Director: Robert R. Rath 
Deputy Executive Director: 

Ethel Simon-McWiltiams 
Region Served: Alaska, Idaho. Oregon, 

Montana, and Washington 
OERI Liaison: John Coulson 
(202) 2V4-2133 

Pacific Regional Educational 
Laboratory (PREL) 

1164 Bishop StreeS, Suite 1409 
Honolulu, HI 96813 
(808) 532-1900 

Board Chair: Steve Umetaro (Acting) 
Executive Director: John W. Kofei 
Region Served: American Samoa. 
Commonwealth of the Northern Mariana 
Islands, Federated States of 
Micronesia, Guam, Hawaii, Republic of 
the Marshall Islands, and Republic of 
Paiau 

OERI Liaison: Joseph Wilkes 
(202) 219-2186 

Regional Laboratory for 
Educational Improvement of the 
Northeast and Islands 

300 Brickstone Square, Suite 900 

Andover, MA 01810 

(508) 470-0098 

Board Chair: Edward Re«dy 
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Executive Director: David P. CrandaK 

Region Served: Connecticut. Maine 
Massachusetts, New Hampshire.' 
New York, Rhode Island, Vermont, 
Puerto Rico, and the Virgin Islands 

OERI Liaison: John C. Egermeier 

(202) 219-2119 

Research for Better Schools, Inc 
(RBS) 

444 North Third Street 

Philadelphia, PA 19123-4107 

(215)574-9300 

Board Chair: Lozelle DeLuz 

Executive Director: John E, Hopkins 

Deputy Director John A. Connolly 

Region Served: Delaware. Maryland, New 

Jersey, Pennsylvania, and the District of 

Columbia 
OERI Liaison: Susan K.Talley 
(202) 219-2129 



SouthEastern Regional Vision for 
Education (SERVE) 

Headquarters: 

University of North Carolina at Greensboro 
P.O. Box 5367 

Greensboro, NC 27435-5367 
(919) 334-3211 or (800) 755-3277 
Held Office: 

Office of Policy Research and 

Improvement 
Florida Department of Education 
325 West Gaines Street, Suite 414 
Tallahassee, FL 32399-0400 
(904) 488-1611 
Board Chair: Go/. Guy Hunt 
Executive Director: Roy H. Forbes 
Deputy Executive Director. Headquarters: 

Ernest K. Nicholson 
Deputy Executive Director. Reld Office: 

Dorothy K. Routh 
Region Served: Alabama. Florida. 

Georgia, Mississippi. North Carolina. 

and South Carolina 
OERI Liaison: Cheryl Garnette 
(202) 219-2267 

Southwest Educational 
Development Laboratory (SEDL) 

211 East Seventh Street 

Austin, TX 78701 

(512) 47S-S861 

Board Chair: Ed Harris 

Executive Director: Preston C. KronKesky 

Region Served: Arkansas, Louisiana. New 

Mexico. Oklahoma, and Texas 
OERI Liaison: Richard A. Lallmang 
(202) 219-2274 
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25 Educational Research and Development Centers 



To help strengthen student learning in 
the United States, OERI supports 25 
university-based national educational 
research and development centers. The 
office recently established 17 new cen- 
ters following a nationwide grant com- 
petition.The new centers, plus 8 existing 
ones, conduct research on topics that 
will help policymakers, practitioners, 
and parents meet the national education 
goals by the year 2000. In addition to 
addressing specific topics, most also 
will focus on children at risk. Many also 
are collaborating with other universities, 
and many work with elementary and 
secondary schools. All have been di- 
rected by OERI to make sure the infor- 
mation they produce reaches parents, 
teachers, and others who can use it to 
make meaningful changes in America's 
schools. The 25 centers and their col- 
laborating partners are: 

Center on Families, Communities, 
Schools, and Children's Learning 

Boston University 
605 Commonwealth Avenue 
Boston, MA 02215 
(617) 353-3309 

Co-directors: Dr. Don Davies and 

Dr. Joyce Ostein 
Affiliated Organizations: 

Institute for Responsive Education, 
Boston 

Johns Hopkins University 

University of Illinois at Urbana- 
Champaign 

Wheelock College, Boston 

Yale University 
OERI Liaison: Harold Hlmmelfarb 
(202) 219-2223 

National Research Center on 
Assessment, Evaluation, and 
Testing 

University of California at Los Angeles 
Center for the Study of Evaluation 
145 Moore Hall 
Los Angeles, CA 90024-1522 
(213) 206-1530 

Co-directors: Dr. Eva Baker and 
Dr. Robert L Linn 

Affiliated Organizations: 
University of Colorado 
RAND Corporation. Washington. DC 
National Opinion Research Center. 

University of Chicago 
Learning Resource Development 
Center. University of Pittsburgh 

OERI Liaison: David Sweet 

(202) 219-2021 



Center for Research on Effective 
Schooling for Disadvantaged 
Students 

Johns Hopkins University 
3505 North Charles Street 
Baltimore, MD 21218 
(301)338-7570 

Director: Dr. Jomills H. Braddock II 
Affiliated Organizations: 

University of CaMfomia at Santa Barbara 

Northern Arizona University 

Teachers College, Columbia University 

Council of Chief State School Officers, 
Washington, DC 
OERI Liaison: Harold Hlmmelfarb 
(202) 219-2223 
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National Research Center on the 
Gifted and Talented 

University of Connecticut at Storrs 

Department of Educational Psychology 

Storrs, CT 06269-2007 

(203) 486-5279 

Director: Dr. Joseph Renzulli 

Affiliated Organizations: 

University of Georgia 

University of Virginia 

Yale University 
OERI Liaison: Ivor Prftchard 
(202) 219-2223 

National Research Center on 
Cultural Diversity and Second 
Language Learning 

University of California at Santa Cruz 
Kerr Hall 

Santa Cruz, CA 95064 
(408) 459-3501 

Co-directors: Dr. Eugene Garcia and 

Dr. Barry McLaughlin 
Affiliated Organizations: 
Linguistic Minority Research Project of 

the University of California 
Center for Applied Linguistics, 
Washington, DC 
OERI Liaison: Rene Gonzalez 
(202)219-2207 

National Center for Research on 
Educational Accountability ana 
Teacher Evaluation 

Western Michigan University 

401 B. Ellsworth Hall 

Kalamazoo, Ml 49008 

(616) 387-5895 

Director: Dr. Daniel Stufflebeam 

Affiliated Organizations: 
University of Alabama at Tuscaloosa 
College of William and Mary 
University of South Florida 

OERI Liaison: Susan Klein 

(202) 219-2207 
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National Center for Research on 
Teacher Learning 

Michigan State University 
College of Education 
116 Erickson Hall 
East Lansing, Ml 48824-1034 
(517)355-9302 
Director: Dr. Mary Kennedy 
Aftiiated Organizations: 

University of Wisconsin at Madison 

Education Matters, Inc., Boston 
OERI Liaison: Joyce Murphy 
(202)21^-2207 

National Center on Postsecondary 
Teaching, Learning, and 
Assessment 

Pennsylvania State University 

Center for the Study of Higher Education 

403 S. Allen Street, Suite 104 

University Park, PA 16801-5202 

(814) 865-5917 

Director Dr. James L Ratciiff 

Affiliated Organizations: 

University of Illinois at Chicago 

Syracuse University 

Northwestern University 

University of Tennessee at Knoxville 

North Carolina State University 
at Raleigh 

Educational Testi ng Se rvice, 4tm 
Princeton, NJ 

Ithaca College 

Stanford University 
OERI Ualson: Jeffrey Gilmore 
(202) 219-2243 

Center for Research on the Context 
of Secondary/ School Teaching 

Stanford University 
School of Education 
CERAS 

Stanford, CA 94305-3084 
(415)723-4972 

Director: Dr. Milbrey W. McLaughlin 
Affiliated Organizations: 

Michigan State University 
OERI Llatoon: Elizabeth Demarest 
(202) 219--2207 

National Center on Education in 
the Inner Cities 

Temple University 
Center for Research in Human 
Development and Education 
933 Rftter Hall Annex 
13th Street and Cecil B. Moore Avenue 
Philadelphia, PA 19122 
(215)787-3001 

Director: Dr. Margaret C. Wang 
Affiliated Organizations: 

University of Illinois at Chicago 

University of Houston 
OERI Liaison: Oliver Moles 
(202) 219-2207 
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b National Research Center on 
tStudent Learning 

University of Pittsburgh 
Learning Research and 

Development Center 
3939 O'Hara Street 
Pittsburgh, PA 15260 
(412) 624-7450 
Co-directors: Dr. Robert Glaser, 

Or. Lauren Resnick, and Dr. James Voss 
Affiliated Organizations: 

None 

OERI Liaison: Judith Segal 
(202) 210-2021 

Center for the Learning and 
Teaching of Elementary Subjects 

Michigan State University 
College of Education 
East Lansing, Ml 48824 
(517)353-^6470 

Co-directors: Dr. Jere Brophy and 

Dr. Penelope L Peterson 
Affiliated Organizations: 

None 

OERI Liaison: Clara Lawson-Copland 
(202) 210-2021 

National Center for Science 
Teaching and Learning 

fohio State University 
Room 104, Research Center 
1314 Wnnear Road 
Columbus, OH 43212 
(614) 292-3339 
Director: Dr. Arthur L White 
Affiliated Organizations: 
None 

OERI Liaison: Wanda Chambers 
(202) 219-2021 

National Center for Research In 
Mathematical Sciences Education 

University of Wisconsin at Madison 

Wisconsin Center for Education Research 

1025 West Johnson Street 

Madison, Wl 53706 

(608) 263-4285 

Director: Dr. Thomas Romberg 

Affiliated Organizations: 

Harvard University 

San Diego State University 
OERI Liaison: Kent Vlehoever 
(202) 219-2021 

Center for Technology in Education 

Bank Street College of Education 
610 West 112th Street 
New York, NY 10025 
(212) 222-6700 

Acting Director: Dr. Jan Hawkins 



Affiliated Organizations: 
Boft, Beranek, & Newman, Inc., 

Cambridge, MA 
Brown University 
Harvard University 

OERI Liaison: Ram Singh 

(202) 219-2021 

National Center on Adult Literacy 

University of Pennsylvania 

Graduate School of Education 

3700 Walnut Street 

Philadelphia, PA 191 04-621 6 

(215)898-2100 

Director. Dr. Daniel Wagner 

Affifiated Organizations: 

National Center tor Family Literacy 
Louisville, KY 

Center for Applied Linguistics, 
Washington, DC 

City University of New York 

Educational Testing Service, 
Princeton, NJ 

Indiana University 

Johns Hopkins University 

Northwest Regional Laboratory, 
Portland, OR 

University of California at Berkeley 

University of California at Santa Barbara 

University of Delaware 

University of Pittsburgh 
<$feRI Ualson: Ann Benjamin 
(202) 219-2223 

National Center on the EducationaT 
Quality of the Workforce 

University of Pennsylvania 

Institute for Research on Higher Education 

4200 Pine Street 

Philadelphia, PA 191 04-4090 

(215) i 898-4585 

Correctors: Dr. Robert M. Zemsky and 

Dr. Peter Cappelli 
Affiliated Organizations: 

Wharton School, University of 
Pennsylvania 

Cornell University 
OERI Ualson: Nevzer Stacey 
(202) 219-2243 



National Center for the Study of 
Writing and Literacy 

University of California at Berkeley 
School of Education 
5513Tolman Hall 
Berkeley, CA 94720 
(415) 643-7022 

Director Dr. Sarah W. Freedman 
Affiliated Organizations: 

Carnegie Mellon University 
OERI Liaison: Steven Hunt 
(202) 219-2243 



National Research Center on 
Literature Teaching and Learning 

State University of New York at Albany 
School of Education 
1400 Washington Avenue 
Albany, NY 12222 
(518) 442-5026 
Director Dr. Arthur N. Applebee 
Affiliated Organizations: 
None 

OERI Liaison: Rita Foy 
(202)219-2021 

Reading Research and Education 
Center 

University of Illinois 

174 Children* Research Center 

51 Gerty Drive 

Champaign, IL61820 

(217)333-2552 

Director Dr, Richard C.Anderson 
Affiliated Organizations: 
Bolt, Beranek and Newman, Inc., 
Cambridge, MA 
OERI Ualson: Anne Sweet 
(202)219-2021 



Center on Organization and 
Restructuring of Schools 

University of Wisconsin at Madison 

Wisconsin Center for Education Research 

1025 West Johnson Street 

Madison, Wl 53706 

(608) 263-7575 

Director: Dr. Fred M. Newmann 

Affifiated Organizations: 

University of Minnesota 

Harvard University 

University of Pennsylvania 

Stanford University 

University of Chicago 
OERI Liaison: David Stevenson 
(202) 219-2207 

The Policy Center of the 
Consortium for Policy Research in 
Education 

Eagleton Institute of Politics 
Rutgers University 
Wood Lawn Neilson Campus 
Clifton Avenue 

New Brunswick, NJ 08903-0270 
(908) 828-3872 
Director: Dr. Susan Fuhrman 
Affiliated Organizations: 

Michigan State University 

Stanford University 

University of Wisconsin at Madison 

Harvard University (year 2) 
OERI Liaison: James Fox 
(202) 219-2223 
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The Finance Center of the 
Consortium for Policy Research in 
Education 

University of Southern California 

School of Education 

Waite Phillips Hali 901 

bos Angeles, CA 90089-0031 

(213)740-3299 

Co-directors: Or. Allan Odden and 

Or. Susan Fuhrman 
Affiliated Organizations: 

Rutgers University 

Michigan State University 

Stanford University 

University of Wisconsin at Madison 

Harvard University 

Cornell University 

SM8 Economics, Washington, DC 



Educational Testing Service, 
Princeton, NJ 
OERI Liaison: Duc-LeTo 
(202) 219-2243 



National Center for Educational 
Leadership 

Harvard University 
Gutman Library 
6 Appian Way 

Cambridge, MA 02138-3704 
(617)495-3575 

Co-directors: Dr. Lee G. Boiman and 

Dr. Terr once E Deal 
Affiliated Organizations: 

Vanderbilt University 

University of Chicago 
OERI Liaison: Ron Anson 
(202) 219-2207 



National Center for School 
Leadership 




University of Illinois at Urbana-Champaign 
College of Education 
1208 West Springfield Avenue 
Urbana, IL61801 

(217) 244-1122 or (800) 356-0069 

Director Dr. Paul W. Thurston 

Affifated Organizations: 
University of Michigan 
Metritech, Inc. at Urban a, IL 
Illinois State Board of Education 

OERI Liaison: Ron Anson, 

(202)219-2207 
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APPENDIX C 



Reply to Comments of Mary Kennedy, 
Chair, Organization of Research Centers (ORC) 
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Reactions to Mary Kennedy's Letter 
About the Draft Report on Centers 
May 20, 1993 



Mary Kennedy, as Chair of the Organization of Research Centers 
(ORC), wrote a four-page critique of the draft reports on the 
quality of work produced by the OERI research and development 
centers (copy of her letter follows) . I telephoned Mary Kennedy 
and thanked her for the comments as well as indicated where and why 
we still disagree on some items. As with the other comments about 
my draft center and lab reports, her specific criticisms have been 
addressed directly in the revised center and laboratory reports. 
However, it might be useful for the reader to have my specific 
reactions to the points raised in her letter on behalf of the 
center directors as it illustrates some of the basic differences in 
how the original draft report was perceived and reported. 

Quality of Research 

The most basic criticism that Mary Kennedy makes is that the 
report employs an extremely narrow view of quality—focused almost 
entirely on sample size. As she put it in her letter, "you seem to 
be using a very narrow definition of -quality' which appears to 
consist almost exclusively or sample size, a criterion that only 
applies when statistical analyses are employed." She repeated her 
criticism in an Education Week interview and added that one if you 
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,f [a]sk a researcher to define for you what makes a study good, 
you 1 11 get a half-dozen criteria, none of them sample size." 

The issue of what constitutes quality is an important one, but 
Mary Kennedy's characterization of my definition being based only 
upon sample size is seriously inaccurate and misleading* The draft 
text on the centers makes it abundantly clear that I have looked at 
a variety of factors, not just sample size. Let me illustrate this 
by commenting on each of the six the principles mentioned in her 
letter: 

(1) & (2) The first two principles state that "(a) good 
research addresses question (s) that have theoretical or 
practical significance or both; (b) good research recognizes 
and builds upon other research and argument in the field," 

I certainly would agree with her first two statements. 
Indeed, throughout the report I emphasize the importance of 
doing a careful, critical review of the existing literature as 
part of the initial research design (in fact, I criticize many 
of the center research proposals for failing to have 
interacted sufficiently with the broader research community) • 
The report then goes on to criticize many of the of the 
centers for lacking a coherent and cumulative research 
strategy. Unfortunately, Mary Kennedy neither acknowledged my 
discussions about the broader theoretical and field concerns 
nor addressed my criticisms of some of the centers for failing 
to have a broader and more ongoing research strategy (a 
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serious omission given the original and often restated mission 
of the centers) . 



(3) The third principle states that " (c) good research 
is systematic and there are opportunities to find 
disconf irming evidence. " Again, I strongly agree with this 
principle and have devoted much attention to this issue • For 
example, the draft report pointed out the need for 
representative samples whenever possible and the importance of 
having adequate controls in order to test the impact of 
interventions. It criticized poorly designed studies, such as 
the comparison of Florida and Wisconsin teachers, because they 
made any systematic analyses difficult If not impossible. 

(4) The statement that "good research uses methods for 
measuring or documenting phenomena that are as valid as 
possible" is self-evident among almost all researchers — though 
sometimes there is disagreement about the specific measures 
employed. Indeed, one of the relative strengths of some of 
the center work was their emphasis on documenting or 
describing phenomena — though sometimes at the expense of more 
rigorous analysis. 

(5) The fifth principle is that "good research uses 
research methods that are appropriate to the questions (s) 
posed." Again, no one would argue with this proposition. 
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This matter was explored in several parts of the draft report- 
s-including a discussion of the appropriate and inappropriate 
use of case studies. Similarly , the discussion of the use of 
simple descriptive statistics rather than more sophisticated 
and rigorous analytic methods are noted for several of the 
center studies. 

(6) Finally, "good research reports are explicit about 
definitions and methods so that others can know exactly how 
conclusions were reached." So true. In fact, the draft 
report criticized studies which did not provide information 
about the number of subjects or failed to include the tables 
from which the conclusions were drawn. Moreover, I pointed 
out the unwillingness of many of the centers to provide 
adequately detailed research designs which then could have 
been scrutinized and criticized by reviewers and/or center 
monitors. 

Far from concentrating only on sample sizes, I have addressed 
a broad array of concerns — including those which Mary Kennedy has 
listed as some of her principles of good research. Moreover, her 
casual dismissal of the importance of sample size is troublesome — 
especially since a significant number of center studies were 
negatively affected by this consideration. 

If one looks through the text and footnotes of my draft 
report, it is clear that I was qu te aware of the current scholarly 
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writings on research and have used them extensively throughout my 
work. Her apparent effort to characterize my analysis as being 
based on an almost exclusive focus on sample size as a criterion of 
quality is simply inaccurate and misleading. 

Generalizability of Research 

The second major criticism that Mary Kennedy makes of my work 
is that I fail to address the issue of the generalizability of 
research. As she puts it, " [w]e assume that one reason you focus 
on sample size is that you assume large samples make research 
general izable. 11 She goes on to emphasize the need to take into 
consideration the context and particular setting of the study — 
regardless of the sample size. 

Again, Mary Kennedy has simply misread my draft report. 
Throughout the report I stressed the importance of knowing the 
setting and context of any study and warned readers of the dangers 
of generalizing from a non-representative sample. For example, 
questions about the representativeness of the case studies done by 
the Berkeley Writing Center were raised as well as the use of 
convenience samples by the UCLA Center. Similarly, the 
generalizability of results from the volunteer teachers in the 
Pittsburgh Center math study was questioned. A third example is 
the failure to specify that a certain study was done on a Catholic 
school was criticized because readers might not appreciate the 
potential dangers of using the results from this study uncritically 
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in order to understand behavior in the public schools. 

Mary Kennedy should have been particularly aware of my concern 
and understanding of the limits of generalizing from individual 
case studies. Although the National Center for Research on Teacher 
Learning was not one of the centers I investigated, Mary Kennedy 
visited me on July 29, 1992 in Washington, DC (with Joyce Murphy, 
her OERI center monitor) . We discussed a study of an undergraduate 
honors history course at Michigan State which her center was 
funding (I had not seen the study, but she described it for me). 
I cautioned her about making any generalizations from an honors 
history course at one particular institution and suggested the need 
for replicating that investigation in other courses and at 
different colleges and universities. She strongly disagreed with 
me and argued that the results of this particular analysis would be 
equally applicable even in other settings — such as Eastern Michigan 
University or the University of Michigan. We never resolved our 
disagreement. However, after she left, Joyce Murphy provided me 
with a copy of that study. G. Williamson McDiarmid, the author of 
the case study, wrote that 11 [obviously, case studies of this type 
do not produce generalizable findings." Thus, far from ignoring 
the issue of context and setting, I have dealt with it both in my 
report and in my discussions with the directors of the centers. 

Difference Between "Basic" and "Applied" Research 

Mary Kennedy writes that in our distant past the research 
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community worried about the relative proportion of research 
expenditures that went to basic versus applied research, but this 
distinction has been virtually abandoned today. To raise the issue 
now, she claims "is more than frivolous; it threatens to move us 
backward, to open old wounds, and to distract us from more 
important issues regarding the substance of our work." 

Throughout the report I have acknowledged that it is difficult 
in practice to separate basic and applied research, but I suspect 
that the debate about that distinction has not been relegated to 
our "distant past?* as Mary Kennedy claims. For example, the 
educational researchers on National Academy of Science panel 
studying the Office of Educational Research and Improvement (OERI) 
in 1992 continued to make a distinction between basic and applied 
research. Indeed, the National Academy report specifically stated 
that: "...the centers should undertake considerable more basic 
research than they do." I found no evidence that anyone accused 
their recommendation or approach of being "more than frivolous." 

Moreover, rather than accepting uncritically the widely used 
distinction between basic and applied research, I offered a way of 
refining and improving that discussion. Borrowing from the recent 
work of the Social Science Research Council, I suggested that we 
might want to broaden our discussion by adding the concept of 
"mission-oriented" basic research. It is interesting to note that 
the Social Science Research Council has not abandoned the 
discussion about basic and applied research (nor has the National 
Science Foundation in its recent discussions on the future 
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direction of their agency) . Unfortunately, Mary Kennedy did not 
acknowledge the considerable extent of my discussion of this 
complex issue in the draft report — but instead reverted in her 
letter to attacking stereotypes prevalent 20-30 years ago. 

Distribution of Center Expenditures 

Given the difficulties of categorizing research, Mary Kennedy 
suggests that it may be counterproductive to examine how center 
funds are appropriated to different areas. Accountability cannot 
be dismissed because of difficulties inherent in the task. We have 
a responsibility to the public to document and analyze how federal 
research funds are expended. If no attempts are made to categorize 
how OERI research funds are spent , policy makers and administrators 
will be handicapped in carrying out one of their basic 
responsibilities. Rather than abandoning the system as a whole, 
our collective responsibility is to try to improve it. In the 
report I suggested that the current PMIS system as well as the new 
TORUS system should be improved. 

Part of the problem is that the center directors and their 
research staff usually have not taken the PMIS system very 
seriously and therefore the data they have reported have been 
suspect. As the report clearly stated, when I went to each of the 
five current centers investigated, they were provided with copies 
of my analysis of their center activities and given an opportunity 
to make any changes or adjustments they thought necessary. 
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Therefore, I think that the results reported in this analysis are 
not as "seriously flawed" as Mary Kennedy argues — though certainly 
there is a need for further refinement of the current PMIS system. 
Indeed, the draft report was careful throughout to caution the 
reader about using these data uncritically. 

I do share, however, Mary Kennedy f s skepticism about the 
specific estimate of basic research done at the "policy center " at 
Rutgers University. As the draft report pointed out, the claim of 
such a high percentage of basic research at the Rutgers University 
Center was based upon a change in the data provided after my visit 
to that institution. As the new chair of the Organization of 
Research Centers (ORC) , Mary Kennedy might work with the other 
center directors and the OERI staff to improve the general 
understanding of and agreement on the terms used. Since Mary 
Kennedy characterizes Susan Fuhrman f s (the director of the Rutgers 
University Center) categorization of basic research as "not 
meaningful," perhaps this may provide a useful opening for a 
further dialogue on this matter, both among the centers and with 
OERI. 

Despite the problems in the measurement of the different types 
of work done by the centers and laboratories, I think the report 
has demonstrated the value of looking at this type of information. 
For example, thanks in part to the categorization of the data, the 
lack of development work by the centers was revealed. This was one 
of the more important findings in my analysis and one that the 
earlier National Academy of Science report did not recognize adequately. 
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Moreover, my efforts to categorize center activities were not 
confined to the nature of the research, but also dealt with center 
expenditures on overhead, administrative costs, and dissemination. 
The distinction between research and non-research expenditures was 
less difficult to make (again, the centers were given an 
opportunity to correct their figures). Thus, another major finding 
of this report was that only about one-half of the federal funds 
went for actual research — something which ma/iy policy makers and 
the public do not realize, but should know. 

How Should OERI Determine the Quality of Research 

Mary Kennedy suggests that a review of the quality of the 
research at centers and laboratories would have been better done by 
a panel than by any single individual scholar. That would have 
been my preference as well — though I doubt that the actual results 
would be much different if a disinterested panel rather than just 
one disinterested scholar had performed this task. 

Having single individuals analyze the centers and laboratories 
has been neither uncommon nor considered unacceptable by either the 
National Institute of Education (NIE) or the Office of Educational 
Research and Improvement (OERI) and their constituencies. For 
example, Francis Chase and Kendall Price each examined the centers 
and laboratories in the past. More recently, Brenda Turnbull of 
Policy Studies Associates has been commissioned to do an indepth 
evaluation of certain aspects of laboratory activities. Moreover, 
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many of these studies did not focus just on some portion of the 
center or laboratory activities, but had to look at them as entire 
entities — a much more difficult and ambitious task than my 
assignment. 

The panel studies of the centers and laboratories which have 
been done did not look in detail at the quality of work of these 
institutions — perhaps in part because of the major commitment of 
time and money such an investigation would require. The National 
Academy of Science, for example, originally was expected to do a 
careful review of the centers and laboratories as part of their 
overall analysis of the Office of Educational Research and 
Improvement (OERI) . Unfortunately, they choose not to pursue that 
part of their mission due to a lack of time and resources. 

While I do not think that a panel of other experts is likely 
to come up with a very different set of conclusions about the 
nature and quality of the research and development at the centers 
and laboratories, I certainly would welcome any further systematic 
inquiries. Moreover, I would suggest adding some other tasks for 
that panel as well — such as systematically looking at the nature 
and quality of the technical assistance provided by the 
laboratories. Therefore, I urge the Organization of Research 
Centers (ORC) and the Council for Educational Development and 
Research (CEDaR) to join forces to encourage the Congress and the 
U.S. Department of Education to provide funds for an independent, 
indepth panel study of their activities. 
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Analysis of Deliverables 



Mary Kennedy questions looking at the deliverables sent to 
OERI as examples of their work — especially since many of these 
unpublished deliverables are refined when they are published. This 
is far from a compelling argument. 

First, some of the materials I criticized were published in 
so-called "prestigious journals with rigorous peer review 
processes. " For example, the study of Florida and Wisconsin 
teachers appeared in Educational Evaluation and Pol icy Analysis. 
Despite the high quality of the journal and the peer review 
process, that research was seriously flawed methodologically. 
Rather than accept uncritically the validity of something just 
because its been published, I suggest that it is better to actually 
read the essay carefully in order to determine its strengths and 
weaknesses. As most scholars know, just because something is 
published in a refereed journal does not necessarily mean it is 
correct . 

Second, an especially disturbing finding from my report was 
that most of my serious methodological criticisms based on the 
unpublished deliverables sent to OERI do not disappear in 
subsequent versions. Many of the deliverables are never published 
and those that are published cannot easily or quickly overcome the 
basic conceptual and methodological weaknesses inherent in the 
original design. Moreover, in my visits to the five centers, I 
tried to discuss my specific criticisms of these studies with the 
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authors and listened to all of their rebuttals. Furthermore, I 
encouraged the centers and laboratories to send me any additional 
materials, including published works, which they thought might be 
appropriate (many did provide such materials to me at the time of 
the visit or after I returned to Washington, DC) • 

Third, Mary Kennedy is slighting the importance of ensuring 
the high quality of deliverables, not just for the purpose of 
accountability of the use of public funds, but also for sustaining 
our intellectual responsibility to other researchers and policy 
makers. Most of the research and development reports are widely 
distributed to policy makers and practitioners by the centers and 
laboratories. Indeed, this is an important component of their 
dissemination activities. If the deliverables are as flawed 
(compared to the eventual published materials) as Mary Kennedy 
argues, then the centers and labs should either warn the readers 
accordingly or only disseminate published materials. A much more 
responsible approach, I think, is to try to improve the quality of 
the deliverables as I think most centers and laboratories are 
striving to do. By downplaying the importance of providing center 
and laboratory deliverables of suitable quality, I think Mary 
Kennedy may be doing many of her colleagues and the system as a 
whole a serious disservice. 

Conclusion 

While I appreciate the critical comments about my draft report 
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from Mary Kennedy, on the whole I am dismayed by the surprisingly 
large number of errors and misleading statements in her letter. I 
strongly urge any interested readers to go back to the original 
draft report on the centers and the laboratories (copies of which 
can be obtained from the Office of Research in OERI) to assess for 
themselves the validity and accuracy of Mary Kennedy *s criticisms 
and my response. Nevertheless, her letter on behalf of the center 
directors provided me with an opportunity to restate my original 
findings and to sharpen some important distinctions. I do regret, 
however, that the center directors did not use tnis opportunity 
also to respond to many of the other fundamental criticisms and 
suggestions for improving their institutions and practices — such as 
increasing the relatively modest percentage of federal research 
money spent on research and development or eliminating the 
fragmentary and non-cumulative nature of many of their studies. 



ERLC 



250 





mm [ ^mWm\ 

NIC 


NATIONAL CENTER FOR RESEARCH ON TEACHER LEARNING 




rIl 





College of Education • 776 Erickson Hall * Michigan State University 
East Lansing, Michigan 48824-1034 
Telephone: 5 7 7-355-9302 FAX 57 7-336-2 795 



April 30, 1993 



Dr. Maris Vinovslds 

Office of Educational Research and Improvement 
U. S. Education Department 
555 New Jersey Avenue 
Washington, DC 20208 



Dear Maris; 

As the newly-elected chair of the Organization of Research Centers, I am responding to the 
draft report you wrote on center and lab research. Our response is based primarily on your 
analysis of centers, but many of the points are probably equally applicable to the lab report. 

The draft report has much to recommend it. Your disciplinary background in history serves you 
well, for instance, in your review of the history of NIE/OERI and its research centers. And you 
have' done a careful and thorough job of capturing many of the difficulties bom OERI staff and 
center researchers face in trying .to promote and conduct high-quality research. The report's 
greatest weaknesses appear in its central focus: defining ihe quality of work under way in 
research centers and laboratories. This is unfortunate, in that the -issue of quality is extremely 
important and deserves thoughtful attention, particularly now, when OERI is being considered 
for reauthorization. In this letter, I want to outline some of the issues your report raised for 
center directors and offer some alternative ways of thinking about these issues. 
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1. What Makes Research "good?" 

The central issue you tried to address was the quality of research conducted in centers. Quality 
is a slippery term in the best of circumstances, and no trivial topic in the field of research, where 
researchers have been debating it publicly for nearly two decades. Much of the argument in the 
field has centered on qualitative versus quantitative data, or on case studies as opposed to 
experiments or surveys. These debates have yielded some important insights about the nature 
of research, the most important of which is that there is no single rule or set of rules for 
defining correct procedures. Instead, a number of principles are used and their relative 
importance differs across settings. Here are some examples of these principles: 

(a) good research addresses question(s) that have theoretical or practical significance or 
both; 

(b) good research recognizes and builds on other research and argument in the field; 
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(c) good research is systematic and there are opportunities to find disconfirming 
evidence; 

(d) good research uses methods for measuring or documenting phenomena that are as 
valid as possible; 

(e) good research uses research methods that are appropriate to the question(s) posed; 
and 

(f) good research reports are explicit about definitions and methods so that others can 
know exactly how conclusions were reached. 

Please do not read this as THE criteria for quality. It is a rough approximation of what we 
perceive as prevailing views in the field. Others would no doubt disagree with some or many of 
these points. The point is that your report gives the impression that you are unaware of die ^ 
extensive literature in this area, for you seem to be using a very narrow definition of quality 
which appears to consist almost exclusively of sample size, a criterion that only applies when 
statistical analyses are employed. 

The question of research quality^ an important one, one that OERI should give serious 
thought to But that thought should- be based on current.thinking in the field and should take 
into account the possibility mat criteria might differ depending on the content being studied and 
the norms of different disciplines. So while we are glad you have raised the issue, we are 
concerned that your criteria are overly narrow and don't reflect the thinking in the field as a 
whole. 

2. What makes research generalizable? 

We assume that one reason you focus on sample size is that you assume large samples make 
research generalizable. But as this debate about quality has ensued, debates about the 
generalizability of research findings have also been lively. We now recognize that numerous 
issues influence the generalizability of research findings. Cronbach has defined generalizability 
as depending not only on the sample of research subjects, but on the nature of the treatment 
and on the nature of the context as well. Some researchers have drawn the distinction between 
generalizability, which is a statistical method of extrapolating from one's data, and transferability, 
which is a judgmental process of extending findings to other settings. This distinction is useful 
not only to qualitative work, such as case studies, but to quantitative studies as well: Most 
experimental studies are done within particular settings, for instance, regardless of the size of 
the sample of students who might participate. The extent to which these studies can be applied 
to other settings is a matter of judgement, not a matter of sample size. 
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3. What is the difference between "basic" and "applied" research? 

In our distant past (that is, around 20-30 years ago), we worried a lot about the relative 
proportion of research expenditures that went to basic versus applied research, and we debated 
the relative merits of each. At that time, basic research was theory driven and tended to be 
conducted in laboratory settings. Applied research, on the other hand, addressed practical 
questions and was done in the field. It was assumed that basic research had to precede applied 
research and that applied research built on basic. Now, the distinction has been virtually 
abandoned. Even theoretical work is conducted in the field, and all of it is applied as well as 
theoretical. 

It has taken us a long time to move beyond this counter-productive distinction and the rivalries 
it engendered. To raise it again now is more than frivolous; it threatens to move us backward, 
to open old wounds, and to distract us from more important issues regarding the substance of 
our work. 

4. How can OERI determine how centers are using their funds? 

You also devote a considerable amount of your report to defining the proportion of funds that 
are used for policy research, evaluation, development, or basic or applied research. But the data 
you use for this analysis come from. Project Information forms and are seriously flawed, thus 
casting doubt on the entire analysis. The fact that a policy research center has more funding 
going to basic research than any other center should provide a clue that the data are not 
meaningful One problem is that the terms are not defined, so that each person filling out the 
form uses his or her own idiosyncratic definitions. Another problem is mat the categories are 
not mutually exclusive, and many studies can be defined in multiple ways. I already noted, for 
instance, that researchers rarely distinguish basic from applied research, and it is not clear how 
anyone could distinguish policy research from applied research. 

It seems to us that the more important question is not how much money is going to these 
vaguely-defined categories, but rather whether the money is being used to address important 
questions. That determination, presumably, was made when the centers were originally 
reviewed. 

5. How should OERI determine the quality of the research it supports? 

In the past, OERI has tended to evaluate research promise at the point of funding rather than 
evaluating the quality of work once it was completed. We commend both you and Diane 
Ravitch for raising the question of the quality of completed work. But if OERI is to pursue this 
question seriously, it needs to pay attention to the procedures it uses for review. Research 
promised traditionally been determined by panels of peer reviewers. A similar process of 
peer review could be used to evaluate completed research. Other methods include responses 
from consumers, analysis of citations, or analysis of awards or other forms of recognition for 
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OERI-sponsored work. What these procedures hold in common is that they recognize the 
importance of the field in detennining quality and they recognize that different fields may value 
different kinds of research and employ different criteria to assess research quality. 

A related issue has to do with which center products should be used to evaluate quality. You 
relied mainly on documents that were submitted to the OERI-those things we call 
"deliverables." But many centers do not consider deliverables as their ultimate products, nor do 
they consider OEM their ultimate audience. Instead, deliverables are intended to keep OERI 
abreast of our progress, thinking, and accomplishments. Most deliverables are substantially 
refined, modified, and customized before they are submitted to their intended audiences. In 
fact, some of the studies you criticized were subsequently published in prestigious journals with 
rigorous peer review processes. 

All of these issues are important ones, and we can thank you and Diane for surfacing them. 
However, we strongly urge you not to publish this report as it currently stands. The narrow 
criteria you have used to define quality and generalizability, the misplaced emphasis on basic 
versus applied research, the poor quality of the data you used to determine the distribution of 
funds across types of research all argue against its release. In addition, the procedural issue of 
who should evaluate quality, and how, is an important one. Publication of this report would 
imply that OERI believes one person's judgement and criteria are adequate to assess the diverse 
array of work being done in the labs and centers. 



Sincerely, 



Mary M. Kennedy 
ORC Chaff, 1993-1994 




MK/lmf 



cc: Center Directors 
Sharon Robinson 
Joe Connaty 
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